Investigating Links from Teacher Knowledge, to Classroom Practice, to Student Learning in the Instructional System of the Middle-School Mathematics Classroom

This article was downloaded by: [Fondren Library, Rice University ]On: 22 November 2014, At: 17:20Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Cognition and InstructionPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/hcgi20

Investigating Links from TeacherKnowledge, to Classroom Practice, toStudent Learning in the InstructionalSystem of the Middle-School MathematicsClassroomNicole Shechtman a , Jeremy Roschelle a , Geneva Haertel a &Jennifer Knudsen aa SRI InternationalPublished online: 05 Jul 2010.

To cite this article: Nicole Shechtman , Jeremy Roschelle , Geneva Haertel & Jennifer Knudsen(2010) Investigating Links from Teacher Knowledge, to Classroom Practice, to Student Learning in theInstructional System of the Middle-School Mathematics Classroom, Cognition and Instruction, 28:3,317-359, DOI: 10.1080/07370008.2010.487961

To link to this article: http://dx.doi.org/10.1080/07370008.2010.487961

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoever orhowsoever caused arising directly or indirectly in connection with, in relation to or arisingout of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &

http://www.tandfonline.com/loi/hcgi20

http://www.tandfonline.com/action/showCitFormats?doi=10.1080/07370008.2010.487961

http://dx.doi.org/10.1080/07370008.2010.487961

Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014

http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/page/terms-and-conditions

COGNITION AND INSTRUCTION, 28(3), 317–359, 2010Copyright C© Taylor & Francis Group, LLCISSN: 0737-0008 print / 1532-690X onlineDOI: 10.1080/07370008.2010.487961

Investigating Links from Teacher Knowledge, to ClassroomPractice, to Student Learning in the Instructional System

of the Middle-School Mathematics Classroom

Nicole Shechtman, Jeremy Roschelle, Geneva Haertel, and Jennifer KnudsenSRI International

Using data collected in 125 seventh-grade and 56 eighth-grade Texas classrooms in the context ofthe “Scaling Up SimCalc” research project in 2005–07, we examined relationships between teachers’mathematics knowledge, teachers’ classroom decision making, and student achievement outcomeson topics of rate, proportionality, and linear function—three important and cognitively demandingprealgebra topics. We found that teachers’ mathematical knowledge was correlated with studentachievement in only one study out of three. We also found a lack of correlations between teachers’mathematical knowledge and critical aspects of instructional decision making. Curriculum and otherlearning resources (e.g., technology, student–student interactions) are clearly important factors forstudent learning in addition to, and in interaction with, teachers’ mathematical knowledge. Ourresults suggest that mathematics knowledge for teaching may have a nonlinear relationship withstudent learning, that those effects may be heavily mediated by other instructional factors, and thatshort-term content knowledge gains in teacher workshops may not persist in classroom instruction.We discuss a need in the field for richer models of how “mathematical knowledge for teaching” worksin the context of complete instructional systems.

INTRODUCTION

Recent educational policy discussions about improved mathematics instruction, commonsensewisdom, and emerging research have focused on the concept of a “highly qualified teacher”(National Mathematics Advisory Panel [NMAP], 2008; No Child Left Behind Act, 2001). Studiesthat analyze teacher qualifications can be appealing because those qualifications can be measuredand managed, and thus such studies can be seen as responding to intense pressure to improvestudents’ mathematics achievement, particularly at the prealgebra level (NMAP, 2008). Indeed,the United States has recently decided to dedicate high-profile economic stimulus money toachieving a more equitable distribution of qualified teachers. However, the question of whatfactors result in well-qualified teachers at the middle-school mathematics level has not beenaddressed. In this article, we focus on one prominent qualification: teachers’ content knowledge of

Correspondence should be addressed to Nicole Shechtman, SRI International, 333 Ravenswood Avenue, Menlo Park,CA 94025. E-mail: [email protected]

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014

318 SHECHTMAN, ROSCHELLE, HAERTEL, AND KNUDSEN

the mathematics they will teach. We examine teacher knowledge in a set of studies in which therewas rigorous alignment of content among curriculum, student assessment, teacher knowledgeassessment, and implementation measures. Our finding of only weak evidence for relationshipsamong teacher knowledge and the other factors, particularly student learning underscores thenecessity to develop theory about the role of teacher knowledge within the complexities of acomplete classroom instructional system.

Mathematics content knowledge is an intuitively obvious qualification for mathematics teach-ing (Ball, Lubienski, & Mewborn, 2001). Who could disagree with the idea that teachers needto understand deeply the mathematics they are teaching in order to be effective? Moreover, bothdetailed case studies (Sowder, Philipp, Armstrong, & Schappelle, 1998) and high-profile inter-national comparison studies support a focus on teachers’ mathematical knowledge (Barber &Mourshed, 2007). In addition, specifying and measuring teachers’ content knowledge is rela-tively straightforward compared to more complicated constructs, such as teachers’ pedagogicalphilosophies, or constructs that require classroom observations, such as teachers’ pedagogicalskill. Other easily measured qualifications, such as having a credential, a master’s degree, or moreexperience do not correlate well with student achievement (Buddin & Zamarro, 2009; Hanushek,1986).

Despite the intuitive appeal of relating teachers’ mathematical knowledge to student outcomes,the empirical support for this “obvious” fact has been surprisingly elusive (Ball et al., 2001,p. 441). To support content knowledge-based qualifications, investigators have recently developedinstruments to measure teachers’ mathematical knowledge and have examined the consequencesof varied levels of teachers’ mathematical knowledge in classroom teaching and learning (Hill,Schilling, & Ball, 2004). Earlier research suffered from the use of more distal variables (e.g.,courses taken, degrees attained, results of basic skills tests) (Hill, Rowan, & Ball, 2005). Ball andcolleagues (Ball, 1990; Ball, Hill, & Bass, 2005; Shulman, 1986) have developed the construct of“mathematical knowledge for teaching” (MKT); that is, mathematical knowledge used to carry outthe work of teaching. Using assessments specifically targeting MKT, researchers’ understandingof the links between teacher knowledge and student achievement is expanding; however, existingstudies still find only weak associations (e.g., Hill et al., 2005).

In a key policy document for mathematics education, the NMAP (2008) connected the intuitiveand research sides and framed the issue as follows:

Research on the relationship between teachers’ mathematical knowledge and students’ achievementconfirms the importance of teachers’ content knowledge. It is self-evident that teachers cannot teachwhat they do not know. However, because most studies have relied on proxies for teachers’ mathemat-ical knowledge (such as teacher certification or courses taken), existing research does not reveal thespecific mathematical knowledge and instructional skill needed for effective teaching, especially atthe elementary and middle school level. Direct assessments of teachers’ actual mathematical knowl-edge provide the strongest indication of a relation between teachers’ content knowledge and theirstudents’ achievement. More precise measures are needed to specify in greater detail the relationshipamong elementary and middle school teachers’ mathematical knowledge, their instructional skill,and students’ learning. (p. xxi)

This article posits the need not only to strengthen measurement but also to refine theory. Wewiden the analytical lens to examine MKT in the context of a complete instructional system in the

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014

TEACHER KNOWLEDGE AND STUDENT LEARNING 319

classroom (Roschelle, Singleton, Sabelli, Pea, & Bransford, 2008), analyzing key componentsto generate conjectures about where MKT is necessary, irrelevant, or even a liability. AlthoughMKT in action (e.g., direct teacher-student interactions, teachers’ scaffolding of activities, teach-ers’ selection of instructional materials) can be critical to learning, we argue that this is just one ofa variety of potential learning resources or opportunities that may exist in the classroom (Cohen,Raudenbush, & Ball, 2003). Other resources may include curriculum materials, interactive repre-sentational technology, student–student participation structures, and students’ own prior attitudesand understandings (Roschelle, Knudsen, & Hegedus, 2009). The qualities of such a system’slearning opportunities and its emphases can vary widely among classrooms for reasons that mayor may not be directly related to the teacher’s MKT. For example, in many cases, teachers do notselect the curriculum that they use. Furthermore, there can be ways in which having higher MKTwithout appropriate pedagogical skill can put teachers at a disadvantage (e.g., they may be morelikely to tell rather than allow students to build mathematical arguments on their own). In light ofthe embeddedness of MKT in such a complex and varied instructional system, it is not surprisingthat detecting a strong or constant connection between teacher knowledge and student learninghas proved elusive.

Using data collected in the context of a series of two large-scale randomized experiments(with one embedded quasi-experiment) collectively termed the “Scaling Up SimCalc” studies(Roschelle et al., in press; Tatar et al., 2008), we examine how teachers’ mathematical knowl-edge relates to instructional and outcome variables. These studies were designed to examinethe impacts of short-term replacement units targeting important middle-school mathematics. Theunits integrated interactive representational technology into curricula that supported student en-gagement with technology and a variety of activity structures (Roschelle et al., 2009). Althoughthe SimCalc studies were not planned specifically to investigate links from teacher knowledgeto student outcomes, they provide an excellent opportunity to examine teacher knowledge ina classroom system that by design included a wide variety of resources for learning (e.g., stu-dent interactions with technology or other students). The studies entailed direct assessment ofteachers’ MKT (using a test modeled on Ball, Hill, and colleagues’ work), a range of classroomimplementation measures, and student outcome data. Thus, post-hoc analyses of the target linkswere possible.

In light of the literature examining relationships between teacher MKT and student achieve-ment, additional advantages of using the Scaling Up SimCalc data are as follows:

• The data set is relatively large, including data from 2,669 seventh-grade students and 825eighth-grade students (in contrast, of the three high-quality studies the NMAP cites, only onestudy had more students).

• The data set spans diverse regions in a large state (Texas) and includes a wide variety ofstudents, teachers, and schools, and thus is suitable for addressing the question of the equitabledistribution of teachers with high content knowledge.

• Whereas other large-scale and high-quality studies focused on elementary school mathematics,the data assessed here focused on teaching prealgebra in middle school and thus can informthe important policy issue of improving prealgebra instruction.

These studies also provide the opportunity to study MKT in a context in which the content ofteacher assessments, student assessments, and the curriculum were all designed and developed in

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


close alignment. Prior research examining links between MKT and student achievement focusedon entire grade levels of knowledge, used standardized tests that were not necessarily closelyaligned with content taught in the curriculum, and did not hold curriculum constant across studyparticipants (e.g., Hill et al., 2005). In contrast, the data in the SimCalc studies were collectedaround specific short-term (3-week) interventions with strong alignment among the curriculumand all measures. Student and teacher assessments were developed in tandem to be highlyproximal both to each other and to the curriculum—a design that increased the likelihood ofdetecting relationships if they were present.

The data collected in conducting the Scaling Up SimCalc studies thus provide an importantopportunity for examining a relatively complete causal chain from teacher knowledge, throughimplementation of instruction, to student achievement in the classroom system in the critical areaof middle-school prealgebra teaching and learning.

This article uses the Scaling Up SimCalc data to address three research questions that explorethe nature of MKT across the participant sample and in the context of the replacement units:

1. What is the range of MKT across the sample, and what teacher and contextual factorsaccount for the variability?

2. Does teachers’ MKT predict student achievement?3. Does teachers’ MKT predict their instructional decisions during classroom implementa-

tion?

In our discussion and conclusion, we will argue that our analysis does not answer these questionsdefinitively but does call attention to the need to refine theory (and not just measurement)in order to provide scientifically valid evidence for the intuitively powerful argument linkingteachers’ mathematical knowledge to their classroom practices and student achievement. Weconclude on the basis of our studies that mathematical knowledge for teaching should be examinedand measured within a full set of components that comprise the dynamics of the classroominstructional system.

TEACHERS’ KNOWLEDGE OF MATHEMATICS: REVIEW OF RESEARCH

The research literature relating teacher characteristics to student achievement spans at least fourdecades (Coleman et al., 1966, cited in the NMAP report). In this literature, three lines of inquiryhave played a role in leading researchers to examine specialized subject-area knowledge forteaching and, consequently, the role of teachers’ knowledge of mathematics in mathematicsteaching.

First, process-product literature posited that teaching behaviors might affect student achieve-ment. Valuable findings of this literature showed that teacher practices (e.g., calling attentionto main ideas, making conceptual linkages explicit) help students to learn more (Brophy &Good, 1986; Duncan & Biddle, 1974; Good & Grouws, 1977). The focus on general teachingbehaviors, however, has been criticized for being atheoretical and lacking specific connectionsto mathematics cognition. Yet, despite these limitations, we found this perspective useful in ouranalysis because it foregrounds the potential role of teachers’ knowledge of mathematics in theprocess of teaching, and the possibility that teachers’ knowledge may matter more when teachers’enact certain processes. When teachers use teaching practices like “making conceptual linkages

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


explicit,” teacher knowledge of mathematics may matter more. Thus, classroom processes maysignificantly mediate the effects of teacher knowledge on student achievement.

A second body of research, educational production functions, examined the relationshipbetween educational resources and student outcomes. Production-function research views educa-tional resources as “inputs” to the educational system and thus examines resources such as studentfamily background and socioeconomic status, district financing, teacher salaries, student–teacherratios, material resources, and teacher and classroom characteristics. For teacher characteristics,the research tends to use proxies (e.g., courses taken, certification, scores on basic skills tests)to investigate the relationship between teacher quality and student learning. The NMAP, citingGoldhaber (2007), reported that a “major finding from this literature was that of all school-relatedfactors, teacher quality dominates effects on student achievement” (Ball et al., 2008, p. 24). Ina sense, the educational production function perspective is the default perspective for examininghow the input of teacher mathematical knowledge affects the output, student achievement. Animportant limitation of this default perspective is that it relies heavily on linear modeling. Thus,an input that has a strong but nonlinear effect on student achievement may appear (in a linearmodel) to have a weak effect on student achievement if the sampled population spans regionsof the nonlinear relationship. Further, if there are hidden dependencies inside the black box ofthe production function (such as the aforementioned potential dependency on teaching process),the production function perspective may not yield compelling explanations for high variabilityin the relationship between inputs and outputs. Both these potential limitations are raised in ourdiscussion and conclusion.

A third area of research, teacher knowledge, examined what teachers need to know aboutsubject-matter content to teach it to students. These researchers emphasize that to be effective,teachers need to have both pure content knowledge, and a firm grasp on how to teach that content.Shulman and his colleagues’ pioneering work investigated what expert teachers already know(e.g., Shulman, 1986). They proposed three categories of teacher subject-matter knowledge:content knowledge, pedagogical content knowledge, and curriculum knowledge. This third areaof research provides considerably more detail linking teaching knowledge to students’ cognitionand learning inside the classroom. For example, pedagogical content knowledge can includeteachers’ knowledge of how to analyze student work for potential student misconceptions andhow to work with students to address those misconceptions (e.g., Fennema et al., 1996). Likewise,curriculum knowledge implies that what teachers need to know may vary with the curriculumthey have available.

This article focuses on the MKT construct, which is an extension of the content knowledgeliterature. In addition, the constructs of pedagogical content knowledge and curriculum knowledgeinform our interpretation of other forms of knowledge that may mediate the linkages from contentknowledge to student outcomes. The idea that teacher knowledge has multiple dimensions willlater guide our interpretation of findings; we will suggest that there are strong pedagogical andcurriculum aspects that mediate any potential link from teachers’ content knowledge to studentlearning.

Our focus on the MKT construct includes a concern with how to best measure it. Looking atevidence from high-quality quantitative research studies, NMAP (2008) reviewed proxy variablesfor teachers’ mathematical knowledge and found that research findings about the impact of thesevariables on student achievement were mixed. In terms of college preparation for teaching, U.S.teachers appear as well-prepared as teachers in other countries (U.S. Department of Education,

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


1996). However, successful completion of a college course may not be good evidence of a thoroughunderstanding of foundational school-level mathematics content (Monk, 1994). Interviews showthat Chinese teachers’ knowledge of elementary school mathematics is deeper than that oftheir American counterparts, despite the higher college degree attainment of American teachers(Ma, 1999). Coursework in American universities typically builds on and extends, but does notrevisit, early mathematics and thus may not sufficiently address the mathematical content thatschool mathematics teachers most need. A logical methodological conclusion is that measures ofteachers’ content knowledge should specifically focus on understanding of the mathematics theywill teach.

NMAP’s (2008) thorough literature search revealed only three high-quality studies that directlyexamined correlations between teacher content knowledge and student achievement gains. Thethree studies examined student learning in elementary school (whereas our research looks atthe relationship between teacher knowledge and student achievement in middle school). Thefirst study, conducted in North Carolina, was based on a 10-year data set of all North Carolinaelementary school teachers and students. Clotfelter, Ladd, and Vigdor (2007) found that teacherswith more experience and higher licensure test scores (which include materials on curriculum,instruction, assessment, and mathematical content), as well as those who were licensed by thestate, had more positive effects on student achievement, and that these effects were particularlystrong in mathematics education. Hill et al. (2005) found that teachers’ mathematical knowledgewas significantly related to student achievement gains in both first and third grades after controllingfor key student- and teacher-level covariates. They used MKT as their measure, thus focusing onthe specific content knowledge that teachers use while teaching. Harbison and Hanushek (1992), inthe context of an 8-year study of academic achievement of fourth-grade students and educationalcosts in northern Brazil, examined the effect on student achievement of teachers’ mathematicstest scores. They found a positive correlation between teacher mathematics test scores andstudent achievement in fourth-grade mathematics; at the same time, they also commented onthe “impossibility” of measuring the quality of specific teachers. The Harbison and Hanushek(1992) findings were not significant at the p < .05 level, but NMAP still considered them to benoteworthy.

Across the three studies, the two standardized regression coefficients that were statisticallysignificant were quite low: .05 and .06 (Hill et al., 2005). To place this in context, we note thata meta-analysis of seven studies of teacher effects found that 11% of the total variability instudent achievement gains in mathematics across 1 year of classroom instruction was attributableto teachers (Nye, Konstantopoulos, & Hedges, 2004). Thus it may not be reasonable to expect ahigh correlation of teachers’ mathematics knowledge with student achievement.

In addition to the three studies discussed above, another study the NMAP identified is relevant.Mullens, Murnane, and Willet (1996) analyzed the effects on student achievement of teachers’mathematics knowledge of both basic and advanced mathematics concepts. They found thatteacher test scores on the Belize National Selection Examination did not predict student achieve-ment for basic concepts in mathematics, but did relate to students’ understanding of advancedconcepts. This finding is relevant because the present study used an outcome measure that hadseparate scales for more basic and more advanced concepts.

A number of research groups have developed different types of instruments to capture and reli-ably measure the mathematics knowledge that teachers use in teaching. The Hill et al. (2005) studyused an instrument that was specifically designed by the Study of Instructional Improvement/

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


Learning Mathematics for Teaching project (SII/LMT) at the University of Michigan to measureMKT. Two other instruments are the Diagnostic Teacher Assessments in Mathematics and Science(DTAMS; Bush, 2005 April) developed at the University of Louisville and the Knowledge forAlgebra Teaching (KAT; Floden, McCrory, Ferrini-Mundy, Reckase, & Senk, 2009) developedat Michigan State University. All three instruments are suitable for measuring teacher knowledgein large populations, and their reliability and validity have been established. A fourth instrumentdeveloped at LessonLab has teachers analyze a classroom video to discuss the mathematics thatthey see occurring in student and teacher talk (Kersting, Givvin, Santagata, Sotelo, & Stigler,2009). Because of the varying purposes for the instruments, each has a slightly different designand reflects different conceptualizations of the MKT construct itself. In the present research, wefollowed the SII/LMT project’s approach.

Recent work has also begun to examine another aspect of the chain of influence from teacherknowledge to student achievement: how teachers’ MKT is related to their classroom practice.Recent detailed case studies trace the connection between teachers’ MKT and the “Mathemat-ical Quality of Instruction” (MQI) observed in classroom teaching. Hill and colleagues (2008)concluded that, “Although there is a significant, strong, and positive association between levelsof MKT and the mathematical quality of instruction, we also find that there are important factorsthat mediate this relationship . . . .” (p. 430). In particular, they found that teachers with low MKTwere likely to use imprecise mathematical definitions in class, choose inappropriate supplemen-tary materials, and make more errors. A weakness of their analysis, however, is that it ties the“mathematical quality of instruction” to the teacher’s contributions, with relatively less attentionto the contributions of students and materials. The cases also seem to emphasize situations inwhich teachers have been provided with inadequate or disorganized curricular materials andmust make decisions about how to assemble them. Results might differ with highly organizedcurricular materials. In situations where the materials are more highly structured or students havesignificant opportunities to interact with a computer, opportunities to learn may be distributedacross interactions with the teacher, other students, and digital materials, potentially diminishingthe proportion of the effect of the teacher relative to all effective components of the classroomsystem. Finally, because the case studies present no data about student achievement, it is stillunknown if the link from MKT to MQI also resulted in higher student achievement.

Overall, despite the intuitive appeal of reasoning that connects teacher quality to teachers’MKT, and connects teachers’ MKT to instructional quality and student achievement gains, theliterature has significant gaps. Few statistically significant findings from high-quality studies linkMKT and student achievement, few studies measure all three variables (MKT, instruction, andoutcomes), and no studies address middle-school mathematics. The context of the literature withthese gaps sets the stage for our post-hoc analysis of the Scaling Up SimCalc data.

RESEARCH CONTEXT: THE SCALING UP SIMCALC PROJECT

The Scaling Up SimCalc Project, which is documented elsewhere (Roschelle et al., in press;Tatar et al., 2008), implemented three studies (two large-scale randomized experiments withone embedded quasi-experiment) designed to address the broad research question, “Can a widevariety of teachers use an integration of technology, curriculum, and professional development to

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


increase student learning of complex and conceptually difficult mathematics?” The studies wereimplemented in Texas in 2005–07.

The research grew out of more than a decade of research and development of the SimCalctechnology and approach whose mission is to “democratize access to the mathematics of changeand variation” (Kaput, 1994, 1997)—that strand of mathematics that relates to much of algebraand leads to calculus. SimCalc emphasizes the concepts of rate and accumulation as thematiccontent that can be developed across many grade levels. The SimCalc project team believed thatreconceptualizing some of the mathematics in middle school and high school in light of the broadermathematics of change and variation could yield a more coherent and fruitful mathematicalexperience for both disadvantaged and advantaged learners (Kaput, 1997).

The SimCalc MathWorlds software provides a “representational infrastructure” (Kaput, Hege-dus, & Lesh, 2007; Kaput & Roschelle, 1998) that is central to enabling this approach. Mostdistinctively, the software presents animations of motion (Figure 1). Students can control themotions of animated characters by building and editing mathematical functions in either graph-ical or algebraic forms. After editing the functions, students can press a play button to see thecorresponding animation. Functions can be displayed in algebraic, graphical, and tabular form,and students are often asked to tell stories that correspond to the functions (and animations). Thesoftware is meant to be used in what Dewey described as a cycle of “doing and undergoing”(Dewey, 1938).

This software affords the hallmarks of the SimCalc approach to learning the mathematics ofchange and variation by:

1. Anchoring students’ efforts to make sense of conceptually rich mathematics in theirexperience of familiar motions, which are portrayed as computer animations.

2. Engaging students in activities to make and analyze graphs that control animations.3. Introducing piecewise linear functions as models of everyday situations with changing

rates.

FIGURE 1 MathWorlds software links graphs to animations of motion.

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


4. Connecting students’ mathematical understanding of rate and proportionality across keymathematical representations (algebraic expressions, tables, graphs) and familiar repre-sentations (narrative stories and animations of motion).

5. Structuring pedagogy around a cycle that asks students to make predictions, comparetheir predictions with mathematical reality, and explain any differences.

To examine this approach at scale, we developed two interventions for use at the middle-schoollevel. Each consisted of an integration of the SimCalc software, curriculum, and teacher profes-sional development. The interventions entailed 3-week replacement units that used the SimCalcsoftware and targeted central and cross-cutting middle-school mathematics: rate, proportional-ity, and linear function. As part of the intervention, teachers were provided brief professionaldevelopment in the mathematical content, use of the software, and use of the curriculum materials.

Prior to conducting this study, there was no available literature that specifically addressed thepedagogical challenges that teachers would experience when using the materials (the literatureon teachers’ experiences with SimCalc only described successful teachers or insightful teachingmoments). Thus, the potential pedagogical challenges had to be deduced from more generalliterature. The most relevant more general literature is about pedagogical challenges in teachingconceptually rich mathematics. For example, Hiebert and Grouws (2007) argue that teachers mustpresent and focus on connections among mathematics ideas. Obviously, this would be difficultto do if they lacked sufficient MKT. Likewise, they argue that teachers must engage students instruggling with the meaning of mathematical ideas. If teachers do not have sufficient depth ofknowledge or comfort with the ideas, they may be unable to support students in making math-ematical connections. Thus, we conjectured that MKT would be necessary in any mathematicsteaching that includes a strong focus on conceptual understanding.

Note that we did not conjecture that knowledge of or experience with using technology inthe classroom would predict student learning for several reasons. First, the MathWorlds softwarewas designed to be intuitive for even novice computer users. Second, the teacher professionaldevelopment provided thorough training in how to use the software. Third, the SimCalc approachprovided the integration of technology and curriculum so that teachers did not need to makeinstructional decisions about when and how to use technology.

To examine the role of MKT in the implementation of these replacement units, use of theintervention logic model (Figure 2) is important. The major inputs of the project to classroomswere software, written curriculum, and the professional development required to implementthose materials. Hence, the intervention focused on content and materials, not on changing or

FIGURE 2 Scaling Up SimCalc logic model. Adapted from Cohen et al. (2003).

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


varying teacher quality. (Nonetheless, as we indicate later, the sampled teachers did vary in theirmathematical knowledge.) We examined classroom instruction as an interaction among teachers,students, and content, and we considered each of the three relationships to be important. Forexample, the project assumed that students can learn from direct engagement with software andworkbook content. In addition, the project assumed that students learn from interacting withteachers and observing how teachers interact with materials and content (e.g., in a classroomdemonstration).

Because of these interconnecting relationships, results from the Scaling Up SimCalc exper-iment should be interpreted as following from the integration of software, paper workbooks,and teacher professional development in a classroom activity system. Within this systems view,the project measured the relationships shown in Figure 2 separately. For example, investigatorsmeasured how students related to content by collecting their workbooks. Likewise, investigatorsmeasured how teachers related to content by collecting the logs in which teachers reported whatthey did each day in class.

Furthermore, we built on the literature on teaching reform. Ball et al. (2001, p. 437) argue that“interpreting reform ideas, managing the challenges of change, using new curriculum materials,enacting new practices, and teaching new content all depend on teachers’ knowledge of mathe-matics.” One virtue of the Scaling Up SimCalc study is that all these conjectured factors werepresent and thus the conjectured dependency on teachers’ mathematical knowledge should besalient. In particular, teachers using our interventions were asked to use new curriculum materialsthat involved at least one major change: the intensive use of technology. Further, the interventionsemphasized teaching new content: more advanced and cognitively demanding aspects of rate,proportionality, and linear function. This new content is important to students’ preparation foralgebra. The fourth factor, enacting new practices, was somewhat less salient; although the studyintroduced new practices (e.g., making predictions), the investigators did not expect teachers’ tochange their pedagogy substantially and the 1-week professional development summer workshopwas too short to produce sustained change in practices. The other three factors, however, inthe Scaling Up SimCalc study provide useful data for looking for relationships of MKT withinstructional variables and student outcomes.

As described elsewhere and illustrated in Figure 3, in all three studies we found significantlearning gains for students in our treatment groups compared with those in the control groupswho used business-as-usual curricula. Analyses revealed statistically significant main effects,with student-level effect sizes of .63, .50, and .56, which are large for educational research.The greatest learning gains were found on the assessment subscale that targeted more advancedmathematics. With regard to simpler mathematics concepts, students fared equally well witheither the intervention or existing materials. These findings were robust across the wide diversityof student ethnicity and socioeconomic status groups across Texas.

In addition to the quantitative main effects, three teams of investigators conducted case studiesin selected SimCalc classrooms during the experiments. Results from these investigations arebriefly introduced here, and we will return to them in the discussion section as they are useful ininterpreting the present results. Empson, Greenstein, Maldonado, and Roschelle (2009) examinedthree seventh-grade classrooms, looking particularly at the overall “configurations” of learningresources during the use of SimCalc. For example, teacher presentation was emphasized morein some classrooms, while student use of computers was a greater focus in others. In addition,these authors mapped discourse during full classroom discussions, looking at the coherence and

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


FIGURE 3 In the Scaling Up SimCalc project, student mean difference scores (± SE of total using HLM) at the studentlevel. M1 items refer to simpler grade-level content, and M2 items refer to content that is more advanced than what istypically taught. From Rochelle et al. (in press).

connectedness of teacher and student talk around mathematical ideas. They reported on oneteacher with high MKT who had mediocre classroom learning gains, likely because teachingpractices blocked many students’ access to learning resources. In contrast, they reported ontwo teachers with high classroom learning gains, one with higher and one with lower MKT. Theteachers had different “configurations” of learning resources—different emphases in their instruc-tional models. Dunn (2009) examined many eighth-grade classrooms and also found considerablevariation in instructional approaches, problematic situations with “blocked” access to learningresources, and variation in the relationship between teachers’ MKT and classroom learning gains.Furthermore, this author also showed that the SimCalc teacher professional development was notprescriptive; it did not lead teachers to follow particular teaching scripts. Pierson (2008) studiedthe details of classroom discourse for the same lesson across 13 seventh-grade classrooms. Shecharacterized each turn of teacher talk in terms of (1) the teacher’s responsiveness to students’mathematical ideas; and (2) whether the talk entailed “giving” or “demanding” mathematicalideas with either low-level intellectual work (e.g., identify/read values, perform calculation, usefact or definition) or high-level intellectual work (e.g., predict, conjecture, generalize, interpret,connect). She found a strong correlation between teachers’ responsiveness to student ideas andstudent learning, and a correlation between teachers’ MKT and their tendency to give mathemat-ical ideas.

SCALING UP SIMCALC’S CONCEPTUAL FRAMEWORK FOR THEMKT CONSTRUCT

We built on the teacher knowledge literature and prior studies’ approaches to measuring MKT(Ball, 1990; Ball et al., 2005; Hill et al., 2005; Ma, 1999; Shulman, 1986) to develop the project’sconceptual framework for the MKT construct. It is important to note that the nature of teacherknowledge is complex and multifaceted and that different researchers define and use the MKTconstruct in different ways for different purposes. The SimCalc MKT framework was developed

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


TABLE 1In SimCalc’s Conceptual Framework for MKT, Core Types of Knowledge, Skills, and Abilities Teachers Should

Know and be Able to do

1. Link and translate between precise aspects of functional representations (i.e., story, graph, table, algebra)2. Evaluate the validity of students’ mathematical conjectures3. Differentiate between colloquial and mathematical uses of language and evaluate student statements for their

mathematical precision4. Interpret common unconventional (in many cases, mathematically correct) forms or representations that students are

likely to make as they construct their understanding5. Generate, choose, and evaluate problems and examples that can illustrate key curricular ideas6. Make connections to important advanced mathematics beyond the unit

to capture the core types of mathematical knowledge teachers would need to support students’learning of the focal mathematical concepts in the SimCalc units, including (1) knowledge of thoseconcepts, and (2) specialized mathematics knowledge necessary during instruction to evaluatestudent thinking and work and to help students make connections within and across concepts.Our definition of MKT does not include pedagogical knowledge—knowledge of what teachingmoves would support students’ learning. For example, MKT includes the knowledge a teacherneeds to judge the mathematical quality of the typical kinds of work that students do, but doesnot include knowledge of how frequently they should expect to see various kinds of work orknowledge of what are the best ways to respond to particular kinds of work. Thus, MKT isessentially mathematical knowledge, but a specialized type that teachers need to make sense ofstudents’ mathematical work.

The framework entails six specific types of knowledge, skills, and abilities that teachersshould know or be able to do, as outlined in Table 1. As described in the section describing theassessment development process, specific MKT relevant for each curriculum was determined byan expert panel examining the cells of a matrix crossing these seven types of knowledge withthe specific mathematical content covered in the curriculum and student assessments. We did notseek comprehensive coverage of this matrix, but rather used it as a tool to prompt for the variousimportant facets of teacher knowledge.

Item mathematical content and development are described in more detail later, and sampleitems can be found in Appendix A.

RESEARCH DESIGN

The new analyses in this article examine the role of MKT in the implementation of the ScalingUp SimCalc studies. Here, we provide an overview of the research design, research participants,treatment intervention, development of the assessments, other instruments, and approach to dataanalysis. To inform the interpretation of the links between teacher knowledge and student achieve-ment, this overview includes a detailed description of the assessment development processes. Fulldocumentation of the design of the curriculum and research can be found in Roschelle et al. (inpress).

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


Experimental and Quasi-Experimental Design

The overall research design comprised two large-scale randomized experiments with one embed-ded quasi-experiment. The Seventh-Grade Experiment, began in summer 2005 with a focus onseventh-grade content, students, and teachers. The Eighth-Grade Experiment began in summer2006 and was designed to extend the findings of the Seventh-Grade Experiment to eighth-gradecontent, students, and teachers. Teachers were randomly assigned (at the school level) either tothe treatment group or to the control group at the beginning of each study. Treatment teacherswere provided the SimCalc treatment intervention (as described in detail later), and the controlteachers were provided professional development of similar quality but of different focus andasked to teach the mathematical content covered in the SimCalc units using their business-as-usualcurricula. The Seventh-Grade Experiment lasted a second year and followed a delayed-treatmentdesign. The second year of the study allowed an embedded Seventh-Grade Quasi-experiment inwhich control teachers (i.e., the delayed-treatment teachers) were provided the SimCalc treatmentintervention. In this quasi-experimental design, we compared the outcomes of the classrooms ofthe delayed-treatment teachers in year 1 when they implemented their business-as-usual curricu-lum and year 2 when they implemented the SimCalc curriculum. Also in the second year, theimmediate-treatment teachers who used SimCalc in their first year taught with SimCalc a secondtime.

The primary dependent measure in each study was student achievement. In both treatmentand control groups, a pretest was administered at the beginning of the unit and a posttest wasadministered at the end of the unit.

Teacher MKT was measured for both treatment and control teachers at key time pointsover the course of the study. Within each study, identical assessments were administered asfollows:

• Baseline. In both major studies, MKT was measured before any professional development hadtaken place. Assessments were administered at the beginning of the summer workshops in2005 and 2006 for the Seventh- and Eighth-Grade Studies, respectively.

• Postworkshop. In the Seventh-Grade Study only, MKT was measured at the end of the 5-daytreatment and 2-day control summer workshops, which are described in more detail later.One of the focuses of these workshops was the MKT for proportionality, relevant to teachingboth the SimCalc unit and business-as-usual curricula. While the administrations were closein time, this second assessment would afford the examination of potential learning gainsfrom the workshop and potentially differential learning gains across the treatment and controlgroups. Note that since both groups received training in MKT for proportionality, it would beimpossible to conclusively tease apart the impacts of the workshop versus learning from takingthe same assessment two times. However, the treatment workshop focused on additional MKT,connections to more advanced mathematics beyond the unit, for which differential growthacross groups plausibly could be causally attributed to learning in the workshop.

• On completion of the study. In both major experiments, MKT was measured after teacherscompleted teaching their units for the last time: after teaching in year 2 in the Seventh-GradeExperiment, and after teaching in the 1 year in the Eighth-Grade Experiment. This approachallowed examination of growth in knowledge through teaching the materials.

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


Components of the Treatment Interventions

The major components of the treatment—mathematical content, curricula, and teacher profes-sional development—are described in this section. Note that in our experiments we did notevaluate the strength of the contribution of each of these components, but rather the impact of theintervention as a whole.

Focal Mathematical Content. In conjunction with a mathematics advisory board that in-cluded three mathematicians and three mathematics educators, the team developed a mathematicsframework for the curricula and student assessments, as shown in Table 2. The team identifiedproportionality and linear function as the target mathematics for several reasons. Among middle-school mathematical concepts, proportionality ranks high in importance, centrality, and difficulty(Hiebert & Carpenter, 1992; National Council of Teachers of Mathematics, 2000; Post, Cramer,Behr, Lesh, & Harel, 1993). Proportionality is closely related to the important concepts of rate,linearity, slope, and covariation. In addition, proportionality offers an opportunity to introducestudents to the concept of a function, through the constant of proportionality, k, that relates x andf(x) in the functional equations of the form f(x) = kx. A deep understanding of the concept offunction as it relates to rate, linearity, slope, and covariation is central to progress in algebra andcalculus. Mathematics education research has identified persistent difficulties in mastering theseconcepts and has theorized that proportionality is at the heart of the conceptually challenging

TABLE 2Mathematical Conceptual Frameworks for the Seventh-Grade and Eighth-Grade Curricula, Student

Assessments, and MKT Assessments

M1 M2

Overview of concepts Concepts typically covered in thegrade-level standards, curricula, andassessments

Building on the foundations of M1

concepts, conceptual building blocks ofthe mathematics of change andvariation found in algebra, calculus,and the sciences

Seventh-Grade Studies(focus on rate andproportionality)

Simple a/b = c/d or y = kx problems inwhich all but one of the values areprovided, and the last must becalculated; basic graph and tablereading without interpretation (e.g.,given a particular value, finding thecorresponding value in a graph or tableof a relationship)

Reasoning about a representation (e.g.,graph, table, or y = kx formula) inwhich a multiplicative constant krepresents a constant rate, slope, speed,or scaling factor across three or morepairs of values that are given orimplied; reasoning across two or morerepresentations

Eighth-Grade Experiment(focus on linearfunction)

Categorizing functions as linear/nonlinearand proportional/nonproportional;within one representation of one linearfunction (formula, table, graph,narrative), finding an input or outputvalue; translating one linear functionfrom one representation to another

Interpreting two or more functions thatrepresent change over time, includinglinear functions or segments ofpiecewise linear functions; finding theaverage rate over a single multiratepiecewise linear function

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


shift from additive to multiplicative reasoning (Harel & Confrey, 1994; Leinhardt, Zaslavsky, &Stein, 1990; Vergnaud, 1988).

We use the symbol M1 to refer to the mathematics that is measured on the tests used foraccountability in Texas. The mathematics tested in Texas embodies a formula approach to pro-portionality and linearity, and tends to ask students to find a number, given two or three othernumbers. We use the symbol M2 to refer to mathematics that goes beyond what is tested in Texas.This mathematics embodies a function approach to proportionality and linear function, and oftenasks students to consider the mapping between a domain and range and to connect such conceptsas rate across multiple representations (e.g., k, in y = kx and the slope in a graph of y = kx).

Curricula Replacement Units. We designed two replacement units, one for the seventhgrade and one for the eighth grade. Each unit covered the relevant mathematical content asoutlined in Table 1. The materials for both units were student workbooks, a teacher’s guide, andcorresponding SimCalc MathWorlds files; the materials were designed to be used daily over a 2-to 3-week period, replacing regular lessons on the same topics. The computer files configured thesoftware to fit a particular lesson. Teachers were required to have access to computer laboratoriesor classroom computer sets, but students could share computers. Teachers could teach their unitby simply following the problems and questions posed in the workbook in the order given. Thesewere not “scripted” curricula, but they did suggest movements between small group work, wholeclass discussion, and individual seat work. The teacher guides provided lesson plans that teacherscould adapt and hints on possible student responses.

The seventh-grade curriculum, Managing the Soccer Team, addressed central concepts ofproportionality: linear function, in the form y = kx, and rate. Speed as rate was devel-oped through a sequence of increasingly complicated simulations. Lessons progressed throughrepresentations—from graphs, to tables, to equations—until students could translate among allthree and connect each concept to verbal descriptions of motion or other real-world contexts. Theunit’s contextual theme was that students must serve as a soccer team manager—training players,ordering uniforms, planning trips to games, and negotiating their own salaries.

The eighth-grade curriculum, Designing Cell Phone Games, addressed linear function andaverage rate. Linear functions were developed as models of motion and accumulation. Studentslearned to use different representations of these functions for problem solving and to translateamong the representations. Graphical representations enabled students to efficiently solve tradi-tionally difficult word problems about average rate. The unit’s contextual theme was that studentsare designers of electronic games and must use mathematics to make the games functional.

Teacher Professional Development. For each of the studies, teachers were providedprofessional development opportunities of short duration to strengthen their mathematical contentknowledge, learn to use the curriculum materials, and/or plan specifically how to use the materials.In all three studies, in the summer before implementation, treatment teachers attended a 3-dayworkshop introducing the respective SimCalc replacement units, as well as a 1-day workshop inthe early fall in which they made specific plans for how and when to use the SimCalc materialsin their classroom.

In the Seventh-Grade Studies, teachers were provided with explicit professional developmentopportunities for developing their MKT. Before the 3-day SimCalc material workshop, treatment

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


teachers attended a 2-day workshop called TEXTEAMS, which is highly regarded in Texasand developed in prior work by our research partner, the Dana Center at the University ofTexas at Austin. This workshop addressed core MKT for both M1 and M2 concepts (see Table1), providing teachers with a foundation for teaching the concepts in the unit. Control teachersattended this workshop as well, but not the subsequent SimCalc training. In addition, the SimCalctraining covered mathematical content beyond the unit: extending proportional functions torelating position and velocity functions.

Note that the design of the Eighth-Grade Experiment was slightly different, and we did notprovide such a content workshop.

Participants

Appendix B presents the sample characteristics, illustrating the diversity of regions, teacherdemographics, and student demographics.

For each study, we selected teachers from a voluntary applicant pool open to teachers in eightregions of Texas. Through Texas’s Educational Service Centers (regional teacher support centers),we recruited a pool reflecting Texas’s regional, ethnic, and socioeconomic diversity. Teachers wererandomly assigned by school either to the treatment or to the control group. Whereas 7 of the 20geographical regions in Texas participated in the studies, of particular note is the participation ofRegion 1 because of its unique demographic and socioeconomic characteristics. Region 1, whichis in the Rio Grande Valley adjacent to the Mexican border, has one of the highest poverty levelsin the United States and is predominantly Hispanic. Region 1 participated in the Seventh-GradeStudies; however, because of a shift in local circumstances, the region did not participate in theEighth-Grade Study.

Across the two experiments, the treatment and control groups were equivalent on all school-level, teacher-level, and student-level variables, except student ethnicity in the Eighth-GradeExperiment, in which the treatment group had a higher percentage of Hispanic students. Thissmall difference does not pose a significant threat to the internal validity of the study, becausethe student pretest scores were equivalent across the groups, and ethnicity did not significantlypredict student learning in either experiment. Also, in the Eighth-Grade Experiment, the greaternumber of teachers in the treatment group was an artifact of teachers’ scheduling conflicts with theworkshops to which they were assigned. Because teachers were not informed about the workshoptype until the workshop occurred, consequences for randomization and thus experiment validitywere minimal.

For the Seventh-Grade Quasi-experiment, we considered data from only the 30 delayed-treatment teachers who finished both year 1 and year 2. In a quasi-experiment in which participantsare not randomly assigned to treatment groups, the primary internal threat to validity is thepossibility of nonequivalence of groups, which we examined between years at the school level,teacher level, and student level. In examining the school-level and teacher-level characteristicsof the sample, because the same sample of teachers within the same schools was used, the mainsystematic difference was that teachers in year 2 would all have an additional year of experience.In addition, policy-level or society-level influences could cause systematic differences betweenthe year 1 and year 2 cohorts, but we have no evidence of such a shift occurring that could haveinfluenced the study outcome.

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


At the student level, the groups were equivalent with respect to all variables, except gender.Again, this difference did not pose a strong threat to the validity of the study, because the baselineassessment scores were equivalent across the groups, and gender was not shown to significantlypredict student learning.

Design and Development of Student and Teacher Assessments

The student and teacher assessments were developed in tandem using similar processes. Here, weprovide an overview of the development process (see Shechtman, Roschelle, Haertel, & Knudsen,2010, for more detail). To arrive at valid and reliable assessments of both students’ and teachers’mathematical knowledge, the Scaling Up SimCalc team grounded its conceptualization of testconstructs and corresponding test structure and content in the tenets of Evidence Centered Design(ECD; Almond, Steinberg, & Mislevy, 2002; Mislevy, Almond, & Lukas, 2003; Mislevy & Haer-tel, 2007; Mislevy, Steinberg, & Almond, 2002). The ECD framework emphasizes the evidentiarybase for specifying coherent, logical relationships among the following essential assessment el-ements: (1) the complex of knowledge, skills, and abilities (KSAs) that are constituent with theconstruct to be measured; (2) the behaviors or performances that should reveal the target construct;(3) the tasks or situations that should elicit those behaviors; and (4) the rational development ofconstruct-based scoring criteria and rubrics (Messick, 1994). This evidentiary base supports boththe construct and content validity of the assessment. There were several steps, as follows.

Domain Analysis and Modeling. In the first stage of ECD, Domain Analysis and Modeling,the assessment’s conceptual foundation is established. In this phase, experts convene to determinea framework for the assessment, and this framework is then used to establish an assessmentblueprint.

For the student assessments, the focal mathematical content (see Table 2) was developed inconjunction with our advisory board and served as the framework for the student assessment.The blueprints for these assessments had four dimensions: (1) complete coverage of all the M1

and M2 topics, with subscales for each; (2) alignment with Texas state standards; (3) variationin problem contexts (i.e., motion and money); and (4) a diversity of task types (about one-thirdeach of multiple choice, short response, construction of multiple mathematical representations).

For the MKT assessments, the focal content would be the mathematical knowledge necessary toteach the concepts in the student conceptual framework. The blueprint for the MKT assessmenthad only one dimension: within the core types of MKT knowledge (see Table 1), completecoverage of the knowledge necessary to teach the M1 and M2 student topics in Table 2. Followingthe LMT model, to facilitate scoring at large scale, most of the MKT items were multiple-choice. Some of the items probing connections to more advanced mathematics (i.e., connectingposition and velocity functions), which had already been used in prior research would have shortconstructed responses.

Item Development. Using the blueprints as a guide, the team developed a pool of potentialitems for each assessment. For reference, Appendix A provides sample assessment items alignedwith the MKT conceptual framework as outlined in Table 1.

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


For the student assessments, items were drawn from the instrument used in prior SimCalcresearch, standardized tests (The National Assessment of Educational Progress, Trends in In-ternational Mathematics and Science Study, and state tests in Texas and other states), and theresearch literature.

For each MKT assessment, to develop our initial pool of items, we held a 1.5-day item camp,a workshop in which individuals with various types of expertise came together to collaborativelygenerate new assessment items. In addition to the SimCalc curriculum designer and core researchteam, members of the item camps included an experienced middle-school math teacher, matheducation researchers, mathematicians, and assessment experts. Participants were provided withthe SimCalc curriculum, the assessment conceptual frameworks and blueprints for both thestudent and teacher assessments (e.g., Table 1 and Table 2), sample items corresponding to eachof the core types of knowledge in Table 1, and various resources such as the Texas middle-schoolmathematics standards, assessments, and textbooks. They were then asked to use these resourcesto generate items that addressed all the important mathematics teachers should know to supportstudent learning during the unit. In addition, in the seventh-grade assessment, to test mathematicsbeyond the unit we incorporated into the pool items from previous SimCalc research that assessedknowledge of connections between representations of changes in position and velocity.

Item Validation. In item validation, evidence is accumulated to provide a scientifically soundargument that the assessment items do, in fact, measure the constructs they are intended to measureto support the intended interpretation of test scores (American Educational Research Association,American Psychological Association, & National Council on Measurement in Education, 1999).The team used three empirical methods to gather validity evidence.

The first method was expert panel review. The primary concerns of such a review are toevaluate the content and construct validity of the items, to categorize the items so that they can bealigned with the blueprint, and to evaluate the grade-level appropriateness of student items. Eachpanel was comprised of experts in mathematics and mathematics education. For each studentassessment, we conducted both a formative and summative review. In the formative review,experts aligned each item with the Texas standards and our conceptual framework, and providedfeedback about how to make the item grade-level appropriate. In the summative review, conductedafter the items were revised using the data from the formative review and other empirical methods(see later), the panelists made a final alignment of the items with the standards and conceptualframework. For the MKT assessments, after employing the methods described later, two seniormembers of the SimCalc research team conducted a summative review. We asked the experts toalign each item with our content framework and to recommend refinements to enhance clarityand alignment.

The second empirical method, cognitive think-alouds, was used to help clarify the languageand examine the appropriateness of the response processes for the intended construct definition(e.g., for an item intended to assess proportional reasoning, the test-taker used that reasoningand not some other strategy). For each student assessment, we conducted think-aloud interviewswith eight middle-school students representing the full range of achievement levels. For eachMKT assessment, we conducted think-aloud interviews with three teachers known to representa range of MKT levels. For each participant, we documented the time needed for completion,the strategies used, the mathematical mistakes made, difficulties in comprehending the problem

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


due to ambiguous or unclear language, unfamiliar terminology, or confusing calculations (whenapplicable). We then summarized this information for each item, eliminating items that weretoo easy or too difficult, eliminating or modifying items that could be solved successfully byconstruct-irrelevant strategies, and clarifying item text or graphics when necessary.

The third method consisted of field testing with a large sample. Using items from the refineditem pool in conjunction with all accumulated data, we created a prototype test that satisfiedall the constraints of our original blueprint. For the student assessment, we tested the prototypeinstrument with a representative sample of middle-school students (230 for the seventh-gradeassessment and 309 for the eighth-grade assessment). For each MKT assessment we conductedfield testing through a mass mailing to a national random sample of 1,000 middle-school mathe-matics teachers. The response rates were 17.9% and 12.8%, yielding 179 and 128 teachers for theseventh- and eighth-grade assessments, respectively. For both assessments, in regard to key de-mographic variables (gender, age, teaching experience, ethnicity, region type, and first language),the samples were representative of the population of teachers we expected to participate in theScaling Up SimCalc Project; suburban regions, versus rural and urban regions, were slightlyoversampled.

We used classical test theory (CTT) and item response theory (IRT) with the field test data toexamine several critical evidentiary concerns. First, we examined individual items for the rangeof possible responses, statistical variation, ceiling and floor effects, and the capacity of itemsto discriminate among test-takers at different ability levels (using IRT parameters for a two-parameter logistic model). Second, we examined the internal consistency of the relationship oftotal and subscale scores to individual items to the test (i.e., scale reliability). Third, we examinedpossible biases among population subgroups.

Assembly and Documentation. We then used all sources of empirical evidence to eliminateitems that had low discrimination parameters (i.e., items that could not discriminate amongindividuals of differing MKT ability) to select (and occasionally modify) items that were likelyto contribute the most information about a teacher’s MKT ability and to maintain representativecoverage of the assessment conceptual framework. Using the IRT data, we calibrated the MKTassessments to be relatively difficult so that the average score would be about 50%. Items wereassembled to meet the test requirements in the assessment blueprint as closely as possible. Table3 outlines the test specifications.

TABLE 3Summary of Basic Test Specifications for Each Assessment

Whole Form M1 Subscale M2 Subscale

Assessment Items Alpha Items Alpha Items Alpha

Seventh grade (rate and proportionality)Student assessment 30 0.86 11 0.73 19 0.82MKT assessment 24 0.80 — — — —

Eighth grade (linear function)Student assessment 36 0.91 18 0.79 18 0.87MKT assessment 28 0.80 — — — —

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


Other Measures

We also collected data on teacher background and attitudes, school context, and classroomimplementation. Teacher background and attitudes were measured with a survey instrumentadministered at baseline along with the baseline MKT. The survey asked teachers about theirprofessional backgrounds and experience, experience with technology, and their beliefs aboutstudents and themselves as teachers. School-level data were obtained through the Public Edu-cation Information Management System (PEIMS), a publicly available database maintained anddistributed by the Texas Education Agency, the state’s education department.

Classroom implementation was measured through use of a daily log. For each day theytaught the unit, teachers filled out a structured log page probing various instructional decisions.The analyses focused on the following three decisions: (1) topic coverage, as aligned with ourconceptual framework in Table 1, (2) cognitive complexity of learning goals, and (3) use oftechnology. Details about specific questions are described later along with the analyses.

We also measured several other variables, which are reported elsewhere. These includeddemographic data on students, interviews with teachers about their experiences in the program,and qualitative data obtained in interviews with students and classroom observations for use inunderstanding student learning and variations in classroom implementation more fully.

Data Analysis Procedures

The analyses were organized to answer the main research questions, as outlined earlier. Becauseclassrooms were sampled as intact clusters, to model any relationships between MKT and studentachievement we used two-level multilevel models (MLM; see Raudenbush & Bryk, 2002) withstudents nested within classrooms. Analyses that did not specifically address student achievementwere examined by using ordinary least squares (OLS) regression models.

The two-level MLM models were structured as follows. Student baseline or gain score wasthe outcome variable, and MKT was entered as a covariate at the teacher level. In question 2,which examined the relationship between MKT and student gains, we modeled each treatmentgroup separately (rather than together using the treatment group as an indicator variable andinteraction terms). For these analyses, we chose to focus on student gain scores (i.e., the pretestscore, subtracted from the posttest score), rather than predicting the posttest score with the pretestas a covariate. Because the treatment condition was randomly assigned (and therefore expectedto have zero covariance with other predictors), either choice would yield an unbiased estimate ofthe treatment effect. We assigned no particular meaning to ranges of posttest scores and thus hadno particular interest in predicting posttest scores.

The basic model examining the relationship between MKT and student achievement within aparticular treatment group was therefore as follows:

Level 1 (Student): yij = β0j + rij

Level 2 (Teacher): β0j = γ00 + γ01 MKT + u0j

This model may be easier to comprehend in its collapsed form:

yij = γ00 + γ01 MKT + rij + u0j

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


In this model, yji is the gain score of student i in teacher j, MKT is the grand mean centeredteacher MKT score, γ 00 is the expected student gain score when teacher MKT is at mean value,γ 01 is the expected change in gain score per unit change in teacher MKT, rij is the random effectof student i of teacher j, and u0j is the random effect for teacher j. Subsequent analyses examinedcross-level effects (i.e., whether MKT predicts gain differentially by student characteristics) byadding pretest as a random slope at the student level and the interaction between pretest andMKT.

All models were fit using the xtmixed procedure of the Stata version 9 statistical software andrestricted maximum likelihood estimation. Continuous covariates were grand-mean centered.

In each analysis entailing a high number of comparisons, we managed the risk of inflatedType I error rates by using the false discovery rate procedure of Benjamini and Hochberg (1995).This procedure ensures that fewer than 5% of the reported statistically significant results withina logical family of comparisons are due to Type I error.

RESULTS

The results and discussion are organized around the three research questions.

1. What is the Range of MKT Across the Sample, and What Teacher andContextual Factors Account for the Variability?

Table 4 provides the basic descriptive statistics of the MKT scores across studies and experimentalgroup. It is important to note the wide distributions of scores on these assessments, particularlythe pretests. The mean pretest scores were 41.7% and 57.1% on the seventh- and eighth-gradeassessments, respectively (which corresponded well to our IRT calibration in field testing). Onthe seventh-grade assessment, teachers just one standard deviation below the mean got only about5 out of 24 of the items correct. On the eighth-grade assessment, teachers one standard deviationbelow the mean got about 11 out 28 items correct. While a number of teachers scored quite lowon these assessments, a number of teachers’ scores approached 100% correct.

TABLE 4MKT Descriptive Statistics

Pretest Postworkshop Postteaching

Group Max. score n Mean SD n Mean SD n Mean SD

Seventh gradeTreatment 24 47 10.1 4.7 47 13.1 5.8 37 13.6 4.9Control 24 48 9.9 4.4 48 11.6 5.0 30 11.9 5.7

Eighth gradeTreatment 28 33 16.3 5.0 — — — 33 18.1 4.6Control 28 23 16.5 5.1 — — — 23 18.0 4.7

Note. Postworkshop MKT was not measured in the Eighth-Grade Study because MKT was not an emphasis in theteacher professional development.

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


In both studies, t-tests were used to show that pretest MKT did not differ between the groupsfollowing random assignment [Seventh-Grade: t(93) = 0.29, p = 0.77; Eighth-Grade: t(54) =0.14, p = 0.89].

Other Teacher Factors. Using a series of OLS regression analyses, we examined therelationships between teachers’ MKT pretest scores and their professional experience and demo-graphic information. As the top portion of Table 5 shows, none of the teacher variables examinedwas related significantly to the teachers’ MKT.

Contextual Factors: Equitability of Distribution Among Schools in Economically andGeographically Diverse Regions of Texas. In another set of regression analyses, we exam-ined the relationships between teachers’ MKT pretest scores and key contextual variables. As

TABLE 5Regression Models Run in Each Study for Teacher and Context Factors Predicting MKT Pretest

Factor Study Grade Coef. SE Percent Mean SD

Teacher variablesYears teaching mathematics Seventh .032 .063 — 10.3 7.8

Eighth –.12 .089 — 8.9 7.5Has master’s degree Seventh 2.0 1.2 17.9 — —

Eighth –1.6 1.9 14.3 — —Teacher is female Seventh –1.5 1.1 78.9 — —

Eighth 1.9 1.8 83.9 — —Teacher is Hispanic Seventh –1.5 1.1 23.1 — —

Eighth –1.9 2.0 12.5 — —Teacher is White Seventh 1.5 1.1 73.7 — —

Eighth 1.2 1.7 82.2 — —Uses computers in classroom Seventh –.73 1.0 53.4 — —

Eighth .39 1.3 50.0 — —Context variables

Student pretest, M1 subscale Seventh .46 .28 — 7.1 1.7Eighth .22 .29 — 6.8 2.3

Student pretest, M2 subscale Seventh .25 .18 — 5.3 2.6Eighth .15 .26 — 4.5 2.6

Percent qualifying for free lunch Seventh –.032 .017 — 53.3 27.9Eighth .037 .031 — 42.7 21.9

Percent Hispanic in school Seventh –.030 .014 — 47.2 34.2Eighth .060 .029 — 32.9 22.6

Percent White in school Seventh .028 .015 — 45.4 31.9Eighth –.056 .028 — 57.3 23.7

Teacher is in Region 1 Seventh −2.2 1.2 9.5 — —Eighth — — 0 — —

Note. Treatment and control teachers were pooled for total sample sizes of 95 and 56 in the Seventh- and Eighth-GradeStudies, respectively. None of these factors significantly predicted MKT after correction procedures for false discoveryrate.

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


FIGURE 4 Distributions of MKT pretest scores and a socioeconomic indicator. These variables are not significantlycorrelated in either study.

the bottom portion of Table 5 shows, none of the context variables examined was related signifi-cantly to the teachers’ MKT. One potential explanation for this finding could have been a lack ofvariability in MKT or another variable; however, that was not, in fact, the case. For example, asFigure 4 illustrates, teachers’ MKT scores on the pretest and the percent of students in the schoolqualifying for free lunch both spanned a wide range. These findings suggest that teachers’ MKTwas equitably distributed among the economically and geographically diverse regions of Texasin our sample.

Teacher Learning Through Participation in the Project. To examine whether partici-pating in the professional development and/or implementing the unit affected teacher MKT, weexamined whether teacher knowledge grew between time points. In both studies, teachers tookthe same MKT assessment at multiple time points. To investigate the statistical significance of thegrowth over time, we predicted MKT scores using two-level MLM models with time point nestedwithin teacher and indicator variables for time point. As shown in Table 4, in both studies, overallwe found modest growth in knowledge in the Seventh-Grade Study from pretest to postworkshop(z = 7.6, p < .0001) and pretest to postteaching (z = 7.1, p < .0001), and in the Eighth-GradeStudy pretest to postteaching (z = 16.6, p < .0001). To examine differences between groups, weadded indicator variables for condition and time point by condition interaction. In the Seventh-Grade Study, the interaction between group and the postworkshop was significant, indicating thatgrowth was significantly higher for the teachers in the immediate group (z = 2.2, p < .05); theinteraction between group and postteaching was not significant (z = .52, p < .60, n.s.). In theEighth-Grade Study the interaction was not significant (z = .21, p = .84, n.s.), indicating thatgrowth was similar across the two groups and that learning was not likely due to teaching theunit.

To probe further into the differential growth across groups in MKT in the Seventh-Grade Studyfrom pretest to postworkshop, as shown in Figure 5, we examined the individual subscales of

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


FIGURE 5 MKT scores by subscale at three time points in the Seventh-Grade Studies. Note that the timeline is not toscale (i.e., pretest to postworkshop is 5 days and postworkshop to postteaching is about a year and a half).

the assessment. Models predicting just the subcale scores showed that the time point by groupinteraction for postworkshop was significant for the velocity items (z = 3.7, p < .0001) but notthe rate and proportionality items (z = .24, p = .81, n.s.). Velocity was introduced using SimCalcsoftware to the immediate group in the extra workshop days that group attended in the first year.The delayed group did not have access to the SimCalc software or to the extra training until thesecond year. Thus, the strong increase on the velocity subscale from pretest to postworkshop foronly the immediate group is consistent with the groups’ different experiences. It also suggeststhat the causal factor of the growth in knowledge was the workshop, not simply taking the test asecond time. Further, note that we did not administer an MKT assessment for the delayed groupafter its first use of the SimCalc software in the summer workshop a year later. It could be that anequally high postworkshop spike would have occurred for this group if a measurement had beentaken at that point. In any event, both groups had the same level of velocity knowledge at the endof teaching the unit in the second year (i.e., the interaction with the postteaching time point andcondition was not significant); at this point, both groups had all the workshops and at least oneyear of teaching experience with SimCalc.

2. Does Teachers’ Mathematics Knowledge Directly Predict StudentAchievement?

As outlined earlier, to investigate direct relationships between teacher MKT and student learning,we ran several two-level MLM models separately (managing the risk of inflated Type I errorrates by using a false discovery rate procedure), with student gain as the outcome variable andteacher MKT as a predictor. For each of the treatment and control groups (for year 1 and year 2in the quasi-experiment), we examined student gains on the student M1 and M2 subscales (seethe content framework in Table 1) and MKT at each time point.

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


FIGURE 6 Relationship between teacher MKT and student gains aggregated at the classroom level. The only significantrelationship is in the Seventh-Grade Year 1 treatment group. ∗∗p < .01.

The only significant relationship between teacher MKT and student gains was in the Seventh-Grade Study Year 1: in the treatment group, teachers’ MKT pretest was a significant but modestpredictor of student M2 gains (β = 0.13, z = 2.6, p < .01). Figure 6 shows scatterplots of thedata in each study for the MKT pretest and student M2 gains, and Table 6 shows the results ofthe MLM analyses for M2. In the treatment group in the Seventh-Grade Study Year 1, in the

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


TABLE 6Unconditional and Conditional MLMs Predicting M2 Gains

Seventh-Grade Seventh-Grade Eighth-GradeYear 1 Experiment Quasi-Experiment Experiment

Model components Treatment Control Year 2 Year 1 Treatment Control

Unconditional modelIntercept (SE) 4.7 (.24) 1.4 (.21) 3.9 (.31) 1.6 (.27) 4.6 (.30) 1.3 (.27)Variance between 2.1 (18.4%) 1.8 (24.0%) 2.4 (23.5%) 1.8 (23.1%) 2.4 (21.1%) .67 (5.5%)Variance within 9.3 (81.6%) 5.7 (76.0%) 7.8 (76.5%) 6.0 (76.9%) 9.0 (78.9%) 11.5 (94.5%)

Conditional modelIntercept (SE) 3.4 (.54) 1.5 (.54) 3.1 (.80) 1.8 (.72) 3.9 (1.1) 1.9 (.93)MKT (SE) .13∗∗ (.05) –.01 (.05) .09 (.08) –.03 (.07) .04 (.06) –.04 (.05)Variance between 1.8 (16.2%) 1.8 (24.0%) 2.4 (23.5%) 1.9 (24.1%) 2.4 (21.1%) .75 (6.1%)Variance within 9.3 (83.8%) 5.7 (76.0%) 7.8 (76.5%) 6.0 (75.9%) 9.0 (78.9%) 11.5 (93.9%)

∗∗p < .01.

unconditional model 18.4% of the student gain score variance was between teachers, and whenMKT was added as a predictor at the teacher level it accounted for 14.3% of the between-teachervariance. We found no significant relationships in the control group, for M1, or for MKT at othertime points. This finding was not replicated in the Seventh-Grade Quasi-experiment (even thoughit was within the same study), nor in the Eighth-Grade Study. In those two studies, we did notfind that MKT at any time point predicted M1 or M2 student gains significantly.

As implied by the relatively low percent variance explained by MKT, teachers with the highestMKT did not necessarily have the highest student gains. As the scatterplots in Figure 6 reveal,many teachers with low MKT had classrooms with large learning gains, and many teachers withhigh MKT had classrooms with low learning gains. The scatterplots also reveal a distinctivepattern of trends in the comparison groups. In each of these conditions (using teachers’ business-as-usual curriculum materials), the best fit line is slightly negative in slope, and the statisticaltests indicate no significant correlation between teachers’ MKT and student achievement. In eachof the treatment conditions (using SimCalc materials), the best fit line slopes upward, indicatinga trend that higher teacher MKT predicts higher student achievement.

Another possibility was that teacher MKT might predict student gains differently for differenttypes of students. We examined the hypothesis that teachers with higher MKT would have higherlearning gains for students who were at a lower baseline achievement level. In other words,analyses investigated the extent to which there was variation between classrooms in the slope ofthe pretest predicting gains, and the extent to which there was a cross-level effect such that MKTpredicts this slope. In each treatment condition, we ran an MLM predicting M2 gains with threepredictors: pretest treated as a random slope at the student level, MKT at the teacher level, andthe interaction between pretest and MKT. Pretest and MKT were centered so that the interceptscould be interpreted as the expected gain score for mean pretest and mean MKT. Across the threestudies, the only significant effects were that in the Seventh-Grade Year 1 Experiment, consistentwith other findings, MKT independently predicted M2 gains. While the slopes of the pretest-gainsdo vary significantly across teachers, the pretest by MKT interaction was not significant in anystudy (i.e., variation in the pretest-gains slopes could not be accounted for by MKT).

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


3. Does Teachers’ MKT Predict Their Instructional Decisions During ClassroomImplementation?

In this post-hoc exploratory analysis examining the potential relationships between MKT andinstructional decisions, we chose to emphasize those areas of decision making that (1) teachersreported consistently on a daily basis in their log, (2) had been shown to have a significantrelationship with student learning gains in one or more of the studies, and (3) could plausibly beinfluenced by teachers’ MKT. Ideally, if we had started with a stronger instructional model forthe use of SimCalc, it might have been possible to make a priori predictions and develop morespecific measures to test them. At the time the experiments were designed, we did not have asufficiently strong instructional model and thus allowed for considerable variation in instructionand took a more general measurement approach.

A first area of decision making was teachers’ topic focus. It would seem plausible that teacherswith higher MKT would choose to teach a range of topics that included more complex topics,such as those aligned with M2 in Table 1. Further, topic focus is related to student learning viathe idea of opportunity to learn; if a teacher never focuses on a topic, it is unlikely a student willlearn it. In the overall results, we did not find a shift from simple M1 to complex M2 topics, butrather an additive layering; teachers who used SimCalc maintained a focus on simple M1 topicsbut added more complex M2 topics, thus covering a broader spectrum from simple to complexeach instructional day.

A second area of decision making was teachers’ choice of simpler or more complex teachinggoals. The teaching log included five kinds of goals used in a previously validated scale (Porter,2002). Simpler goals were memorization and use of routine procedures. More complex goals werecommunicating conceptual understanding; making mathematical connections; solving nonroutineproblems; and conjecturing, generalizing, or proving. It would seem plausible that teachers withhigher MKT would choose more complex goals and boost students performance on the morecomplex mathematical concepts.

To ascertain each of these instructional decisions, each day in their logs teachers reported thetopics they covered and the teaching goals they emphasized. For topics, the log provided a listaligned with the topics covered in the curriculum and assessment, as outlined in Table 1. Forteaching goals, the log provided the five simple and complex goals described earlier. Teacherswere to rate the extent to which they focused on each topic and goal during the lesson. Teacherschecked boxes on a 4-point Likert scale ranging from (1) not at all, to (4) a major focus. For topicfocus, we considered teachers as focusing on a particular topic in a given day if they selected a3 or 4 on the scale. Teachers were not limited in the number of topics they could rate as a 3 or4. We then counted the number of days a teacher focused on each topic or goal. For goals, wecreated an index for the mean rating across days for the two types of simple goals and an indexfor the mean rating across days for the three types of complex goals.

A third area of decision making was teachers’ choice to spend more or fewer days in thecomputer lab. We chose this decision because use of the computer is critical to the SimCalcintervention and because we wanted to know if MKT might support or interfere with computeruse. We conjectured that use of SimCalc might support computer use because teachers with higherMKT might conclude that the SimCalc software allowed students to engage with more advancedmathematics. Indeed, teachers teaching the SimCalc unit spent much more time in the computerlab than those in the control in both the Seventh-Grade Experiment [an average of 41.5% and

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


TABLE 7Regression Models Run in Each Study for Implementation Variables Predicting MKT Pretest

Factor Study Grade Coefficient SE

Days focused on M1 topics Seventh –.31 .26Eighth .15 .11

Days focused on M2 topics Seventh –.30 .26Eighth .15 .10

Extent of focus on cognitively simple goals Seventh –.02 .02Eighth –.04 .02

Extent of focus on cognitively complex goals Seventh .03 .02Eighth .01 .02

Extent of the use of technology Seventh .21 .12Eighth .08 .13

Note. These are treatment teachers only with sample sizes of 48 and 33 in the Seventh- and Eighth-Grade Studies,respectively. None of these factors significantly predicted MKT after correction procedures for false discovery rate.

3.5% of the days in the treatment and control groups, respectively; t(93) = 8.0, p < .0001] and theEighth-Grade Experiment [an average of 72.8% and 6.6%, respectively; t(54) = 9.4, p < .0001].Further, the amount of time spent in the computer lab was a predictor of student learning gainsin the Eighth-Grade Experiment (z = 2.5, p < .05 for the interaction term of days in computerlab by condition). This finding was not replicated in the Seventh-Grade Experiment (z = 0.24,p = .81, n.s.).

As shown in Table 7, the results supported none of these conjectures. MKT was not statisticallyrelated to any of these implementation variables.

DISCUSSION

Our analysis covered three separate research questions.

Question 1: MKT and Other Factors

The first research question examined relationships between MKT and teacher background andschool context, and examined the extent to which teachers grew in MKT through participation inthe project.

The first finding was that no other teacher factors predicted MKT. This finding concurs withearlier work that argued that the MKT construct is independent of other common measuresof teachers’ background and knowledge. Hill et al. (2005) found that their measures of teach-ers’ mathematical content knowledge were not significantly correlated with teacher preparationor experience variables, with the exception of one weak correlation between certification andmathematics knowledge in grade 3 only.

The second finding was that no contextual variables predict MKT, which implies that MKTwas equitably distributed across the socioeconomically and geographically diverse regions ofTexas. We advise that researchers exercise caution in inferring from our results that MKT is

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


equitably distributed in all populations. Note that this result may not generalize to a broaderpopulation. For example, the Scaling Up SimCalc studies notably lacked large, urban districts,as well as schools in which the majority of students were African American. It could be thatteachers with high MKT are not equitably distributed to and within these or other districts. Tothe extent an equitable distribution of MKT exists, whether it is beneficial for students dependson whether schools can leverage this resource to produce better teaching and learning.

These null results also establish MKT as an independently measured construct that is notcollinear with other variables in our data set. This has important implications for the inferencesthat can be made from the data. For example, if MKT had correlated with other typically influentialvariables (e.g., the percent of students in the school qualifying for free or reduced lunch), it wouldbe more difficult to tease apart the influences of MKT from those of other factors.

A third important finding was that only minimal teacher learning was detected over the courseof their participation in the project, and much of this may be attributable to taking the sametest on repeated occasions. We found that teachers did learn additional mathematics content inthe summer workshop (i.e., connections between position and velocity functions in the Seventh-Grade Study), but that a measurement taken immediately after a workshop may overestimatethe long-term availability of the additional knowledge for teaching. Further, we found that ourworkshop induced the most learning with respect to the novel velocity content, but that thecontent was less relevant to classroom teaching. In general, short-term increases in MKT may notbe stable. Consequently, we emphasized baseline measures of MKT in subsequent analyses andemphasized the rate and proportionality (not velocity) subscale. In the latter findings, the lack ofcorrelation between MKT at later times (e.g., after teachers had learned new material) and studentachievement suggests that teachers’ learning gains were not important to student achievement;indeed, the best predictor was the first MKT measurement, which may represent “stable” teacherknowledge. At the very least, this finding suggests caution in assuming that short-term gains inMKT (e.g., those measured before and after a workshop) will produce meaningful effects duringthe school year.

Question 2: MKT and Student Learning

The overall finding was that MKT predicted student learning gains in M2 in the Seventh-GradeYear 1 Experiment only. This finding was not replicated in the Seventh-Grade Quasi-experiment,even though the sample was a subset of the same teachers in Year 1, nor was it replicated in theEighth-Grade Study. MKT did not predict learning gains in any of the comparison groups, nordid it predict learning gains in any group for M1.

Several explanations for the lack of a consistent correlation between MKT and student achieve-ment are possible. One plausible explanation is that we did not measure MKT sensitively and thata better measurement would have found a stronger linkage. Although this explanation is plausible,we argue that the rigor of our assessment development process and intentional alignment withthe student assessment and curriculum make this explanation unlikely.

A second plausible explanation is that MKT is nonlinearly related to student achievement, andthat all the teachers in our sample reached a high enough threshold. The Scaling Up SimCalcstudies did involve volunteer teachers, and it could be that teachers with insufficient MKT did notvolunteer. This may also explain why the relationship between MKT and student learning was so

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


weak when standard curricula were used; those curricula did not stretch students to understandcognitively demanding mathematics, and all the teachers may have had sufficient MKT for simplermathematics. Further, this may explain why gaining more MKT (e.g., by learning about velocity)might not have been particularly helpful to teachers; it could have been that the new knowledgeoffered was more than needed and did not support the specific content in the unit. This wouldalso be consistent with the findings of Hill et al. (2005), which showed that significant differencesin student achievement occurred only between the teachers ranked on MKT in the lowest 20%and those ranked above that, and that there appeared to be little systematic relationship betweenincreases in MKT and student achievement. If this explanation is true, future investigators ofMKT may want to target populations of teachers with MKT that may be below a particularthreshold for a given topic area.

A variant of this explanation is that the SimCalc intervention was not designed such that teach-ers with higher MKT could engage their knowledge productively. Perhaps to the contrary, in fact,it was designed to be successfully implemented by a wide range of teachers. It is possible that stu-dents would learn even more with higher MKT teachers if the intervention had activity or curricu-lum structures that engaged teachers’ MKT more strongly. If either of these variants is the correctexplanation, future investigators will want to pay close attention to the ways in which teachersuse their MKT in the actions of instruction. For example, we observed that in the Hill et al. (2008)case studies, teachers assembled mathematics materials (and MKT mattered). In the Scaling UpSimCalc studies, teachers did not assemble such materials (and MKT may have mattered less).

We can reject an additional possible explanation that by design the SimCalc interventionwas so tightly scripted that in-depth knowledge of mathematics was simply not required or thatteachers’ felt obligated to avoid typical types of adaptations they ordinarily make when usingnew materials. Case studies of classroom implementations during the experiments (Dunn, 2009;Empson et al., 2009) found large variations in classroom implementations of SimCalc, some thatclearly leveraged teachers’ MKT and others that did not depend so much on it. Clearly SimCalcwas not so scripted as to preclude potentially strong roles for teachers’ mathematics knowledgein choosing different instructional approaches.

A third possible explanation is that the relationship is weak because teachers’ MKT is only oneof a variety of learning resources that were available to students in a classroom system engagedin a technology-integrated curriculum. Our studies found strong main effects of new materialsacross a population of teachers with varied MKT (see Figure 3). It may be that in the presenceof a technology-integrated curriculum, other classroom strengths compensate for weaknesses inMKT. For example, students may be better able and have more time to help each other withmathematics when they have representational tools, and thus be less dependent on the teacher’sdepth of mathematical knowledge. If such compensations exist, even though strong MKT wouldbe beneficial to students, weak MKT would not so strongly deprive students of the opportunityto learn, and therefore the overall strength of the relationship between teacher MKT and studentlearning would be less.

Question 3: MKT and Teacher Instructional Decisions

The overall finding for this research question was that MKT did not correlate with any of the threeareas of instructional decision making we investigated: decisions about topic coverage, choice ofteaching goals, and use of technology.

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


Of course, finding a null result does not “prove” that MKT does not influence teacher deci-sions. It could be that, even though MKT influences topic coverage, choice of more cognitivelydemanding tasks, or the use of computers, our measures did not have the sensitivity to measure theactual effect. At the very least, however, we know that these three decision areas are important tosuccessful implementation of the SimCalc replacement unit. From the limited perspective associ-ated with the replacement unit, we conclude that boosting teacher MKT (or otherwise addressingweak teacher MKT) is unlikely to influence these important implementation decisions. To en-courage teachers to make the right decisions for good implementation of SimCalc replacementunits, we should focus elsewhere.

Strong mathematical understanding may be more important in the more proximal, moment-to-moment classroom teaching activities that require the teacher to work directly with studentthinking, such as those examined in the MQI studies (e.g., questioning, errors, responding tostudents). Furthermore, as indicated by the MQI research, it may only be possible to measuresome important practices that are supported by teacher knowledge through observation, as theymay not be detectable at the degree of granularity of self-reported logs. Studies that use videoanalyses to examine moment-by-moment teaching activities, such as questioning, correctingerrors, and responding to students, show promise for examining in greater depth the affordancesof teachers’ mathematical understanding (Hill et al., 2008; Pierson, 2008). For example, oneset of case studies conducted within the SimCalc project found a potentially interesting link:Pierson (2008) analyzed classroom discourse in a subset of 13 SimCalc classrooms and foundhigher student achievement in classrooms where teachers were more responsive to students’mathematical ideas. Although the extent of teacher questioning was not found to be directlyrelated to their MKT, further research should examine these links in more depth.

However, Pierson’s study also revealed ways in which higher MKT may sometimes be aliability during classroom discussion: teachers with high MKT were more likely to give, notask for, higher-order information. Because giving higher-order information is generally lessproductive for student learning than asking for that information (Hiebert & Grouws, 2007),higher MKT may predispose teachers to suboptimal interactions with students. This finding isconsistent with the two other sets of SimCalc case studies (Empson et al., 2009; Dunn, 2009).Some counterintuitive observations from the findings include: (1) some high-MKT teachers maylimit the time students have to interact directly with high-quality materials (e.g., software);(2) high-MKT teachers may act in more controlling ways, giving students less opportunity toparticipate in mathematical activities; and (3) high-MKT teachers may interpret state standardsto limit focus to simpler mathematical constructs, and thus they may not create opportunities forstudents to learn the more complex mathematics available in the curriculum materials.

These findings underscore the complexity of the relationship between MKT and studentachievement.

CONCLUSION

The Scaling Up SimCalc studies have afforded an examination of the relationships among teach-ers’ MKT, teachers’ instructional decision making, and student achievement in the importantarea of middle-school, prealgebra mathematics within a rich classroom instructional system. Theoperational approach to the MKT construct was based on categories of knowledge identified in

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


the mathematics education literature as comprising both the knowledge of the key curricular con-cepts and the specialized mathematics necessary during instruction to evaluate student thinkingand work, and to help students make connections across concepts. Further, because mathematicsknowledge instruments for teachers and students were aligned with the same framework, thestudies would seem to be a plausible place to look for relationships between teacher knowledgeand student learning. Indeed, the SimCalc setting had three of four factors that Ball, Lubienski,and Mewborn (2001) conjectured as demanding teachers’ mathematical knowledge: providingnew content, implementing reform, and managing change. Yet, we found mixed results.

Specifically, we found inconsistent evidence for a link between MKT and student achievement.Corresponding with commonsense wisdom about teacher knowledge, we found that MKT linkedto student achievement in the treatment condition of our Seventh-Grade Experiment Year One:teachers with higher MKT had students who learned more cognitively demanding mathematics.Furthermore, we found MKT was reasonably equitably distributed in the population we sampledand not linked to other, general-purpose background or context variables. In addition, MKT islearnable; teachers’ knowledge grew in some ways.

However, contrary to conventional intuitions, we did not find correlations between MKT andstudent learning in two of the three studies and did not find correlations at all when teachers usedtheir business-as-usual curriculum materials. Consonant with prior research (i.e., Nye et al., 2004),the percentage of variance at the teacher level was low (roughly 20% in the unconditional modelsacross studies), and even when there was a correlation between MKT and student achievement,it only accounted for less than 15% of the teacher-level variance. In addition, teacher learningof MKT occurred most notably on less relevant, more novel content, and teachers did not retainshort-term learning gains

Our findings suggest that, rather than simply examining the direct relationship between MKTand student achievement, researchers will gain greater insights by investigating how MKT influ-ences student learning within the context of the full classroom instructional system. Through thepresent analyses, the findings of three sets of SimCalc case studies, and a review of the literature,we have identified different types of potential relationships in a given classroom that may exist be-tween MKT and student learning. Examples from this preliminary and nonexhaustive list include:

• Teachers may use their MKT to support student learning through their support of studentthinking in classroom discussion (e.g., questioning), through their facilitation of student en-gagement with specific curricular activities (e.g., scaffolding engagement for mathematicalthoroughness in the workbooks), through using accurate and precise mathematical language,and through the ways they select and structure curriculum materials (which was relevant inthe MQI work, but not the SimCalc studies).

• Higher MKT may hinder teachers’ support of student learning if they use it with a transmittingor controlling approach that limits students’ learning opportunities.

• Teachers’ MKT may be irrelevant to student learning when other learning resources in theclassroom, such as curriculum materials, technology, or student–student interactions, providestrong opportunities to learn.

Given the complexity and variability in the relationships between MKT and student learning, it isnot surprising that it has been an empirical challenge to find simple and direct causal links fromMKT to instructional practice to student learning.

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


It seems premature to make generalizations about the value of MKT as a construct in researchor in policymaking. MKT is clearly a complex construct. It matters, for example, how specificallythe measure of teacher knowledge is tied to the knowledge teachers are likely to use whileteaching. We also need to know more about the stability of MKT and whether teachers’ short-term learning of MKT (e.g., in a summer workshop) has any long-term impact on their teaching.It would be consistent with many other recommendations to focus on teacher development,including development of deeper mathematics knowledge, in the context of long-term and ongoingprofessional development (Darling-Hammond & Sykes, 1999; Garet et al., 1999).

MKT is also likely to be a heavily mediated construct in regard to its effect on student learning,and we do not yet know what the mediating variables are or how they work. It would certainlyseem unwise to expect that MKT has a unitary impact on classroom teaching. High-MKT teachers,for example, may be less responsive to the range of student understandings but better able to givehigh-quality conceptual explanations. Curriculum is likely to be an independent factor that makesMKT more or less relevant; if a curriculum does not require much mathematics knowledge, itseems unlikely that teachers will employ that knowledge, even if they have it. Moreover, teachersmay make fairly major instructional decisions independent of their mathematical knowledge, andwe do not yet know exactly which classroom practices MKT influences most. In addition, ininstructional systems that provide teachers with a lot of structure, teachers may have to makefewer decisions that rely on MKT. In summary, it may be that weak theory (and not weakmeasurement) is the reason that investigators obtain mixed results for the impact of MKT inteaching and learning.

Our principal recommendation for further research is to develop better models of how effectiveinstruction works in order to find the places and ways in which MKT is essential to effectiveinstruction. Concurrent research suggests to us that even holding curriculum constant, teachersenact a wide range of instructional approaches. For example, case studies (Empson et al., 2009)identified one teacher who clearly used MKT to lead well-organized discussions of the connec-tions among mathematical ideas. Another teacher with low MKT also achieved strong classroomlearning gains despite poorly organized full classroom discussions; this teacher supported stu-dents’ small-group and individual work with the materials. In addition, in Hill et al.’s (2008) casestudies, MKT seemed particularly important when teachers had poorly organized instructionalmaterials; the SimCalc materials were comparatively well-organized.

We assert that studies only measuring MKT input and student achievement output with respectto the “black box” of the classroom system are unlikely to lead to further progress in the field.Research that begins with an instructional model (or spectrum of instructional models) is morelikely to trace the causal pathway from teacher MKT to enhanced student learning and can focusmeasurement more tightly around that pathway, which will maximize the possibility of findingstronger correlations than we or other researchers have found. Furthermore, by recognizing andclassifying variation in instructional materials and models, researchers may be able to betterexplain why the effects of teacher MKT are variable across classrooms at the same grade level.

In contrast, we would not recommend that researchers give up on measuring teacher MKT orrelated constructs. This research does not undercut existing arguments for why it is important tomeasure teachers’ mathematics knowledge (e.g., Ball, Hill, & Bass, 2005); these arguments arestill compelling. Instead of pointing at potential invalidity of MKT, we would point at potentialinvalidity of a research approach that considers all classroom instruction to be equal in its use ofMKT. Rather than seeking direct correlations from teachers’ knowledge to student achievement,

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


we would instead focus research attention on the role of teachers’ knowledge in an instructionalmodel that produces student learning. Even further, we would suggest that instruction be seen asa system with multiple potential pathways that affect student learning, and therefore would focuson the role of teachers’ knowledge in classroom systems that produce student learning.

From a policy perspective, our research suggests caution about reducing the quest for “highlyqualified teachers” to selection of teachers for their mathematics content knowledge. We havesuggested that the effect of teachers’ mathematics content knowledge on student learning maybe nonlinear. Consequently, above a certain threshold, other selection variables may be far moreimportant than how much mathematics a teacher knows. In addition, weaker teacher mathematicscontent knowledge may be a variable that can be compensated for in other aspects of the instruc-tional system, such as a better curriculum, use of appropriate instructional technology, or use of apedagogy that engages student thinking at a high level of cognitive demand. Although null resultssometimes fail to excite policymakers, we are not the first to report weak or null results; indeed, theNational Mathematics Advisory Panel had similar findings. It is an important function of researchto alert policymakers to plausible and popular ideas that may not have an empirical basis.

We thus repeat our caution about seeking simple, one-variable solutions to complex educationalproblems. It seems likely that educational improvement requires sophisticated understanding ofthe classroom instructional system and not just the quality of the inputs to that system.

ACKNOWLEDGMENTS

This material is based on work supported by the National Science Foundation under GrantNo. 0437861. Any opinions, findings, and conclusions or recommendations expressed in thismaterial are those of the authors and do not necessarily reflect the views of the National ScienceFoundation. We thank Susan Empson for her insightful comments on this article. We also thankSara Carriere and Larry Gallagher for their data analysis support, as well as Ken Rafanan, PhilVahey, and Gucci Estrella for their support in developing the assessments.

REFERENCES

Almond, R. G., Steinberg, L. S., & Mislevy, R. J. (2002). Enhancing the design and delivery of assessment systems: Afour-process architecture. Journal of Technology, Learning, and Assessment, 1(5).

American Educational Research Association, American Psychological Association, & National Council on Measurementin Education. (1999). Standards for educational and psychological testing. Washington, DC: American EducationalResearch Association.

Ball, D. L. (1990). The mathematical understandings that prospective teachers bring to teacher education. The ElementarySchool Journal, 90(4), 449–466.

Ball, D. L., Hill, H. C., & Bass, H. (2005). Knowing mathematics for teaching. American Educator, 29(3), 14.Ball, D. L., Lubienski, S., & Mewborn, D. S. (2001). Research on teaching mathematics: The unsolved problem of

teachers’ mathematical knowledge. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 433–456).New York: Macmillan.

Ball, D. L., Simons, J., Wu, H.-H., Simon, R., Whitehurst, G., & Yun, J. (2008). Report of the task group on teachers andteacher education. Washington DC: U.S. Department of Education.

Barber, M., & Mourshed, M. (2007). How the world’s best-performing school systems come out on top. San Francisco:McKinsey and Company.

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multipletesting. Journal of the Royal Statistical Society B, 57, 289–300.

Brophy, J., & Good, T. L. (1986). Teacher behavior and student achievement. Handbook of Research on Teaching, 3,328–375.

Buddin, R., & Zamarro, G. (2009). Teacher qualifications and middle school student achievement. Santa Monica CA:RAND.

Bush, W. S. (2005, April). The reliability and validity of diagnostic assessments for middle school teachers. Paperpresented at the annual meeting of American Educational Research Association, Montreal.

Clotfelter, C. T., Ladd, H. F., & Vigdor, J. L. (2007). Teacher credentials and student achievement: Longitudinal analysiswith student fixed effects. Economics of Education Review, 26(6), 673–682.

Cohen, D. K., Raudenbush, S., & Ball, D. L. (2003). Resources, instruction, and research. Educational Evaluation andPolicy Analysis, 25(2), 1–24.

Coleman, J. S., Campbell, E. Q., Hobson, C. J., McPartland, J., Mood, A. M., Weinfeld, F. D., et al. (1966). Equality ofeducational opportunity. Washington DC: U.S. Government Printing Office.

Darling-Hammond, L., & Sykes, G. (1999). Teaching as the learning profession: A handbook of policy and practice. SanFrancisco: Jossey-Bass.

Dewey, J. (1938). Experience and education. New York: Macmillan.Duncan, M. J., & Biddle, B. J. (1974). The study of teaching. New York: Holt, Rinehart, and Winston.Dunn, M. (2009). Investigating variation in teaching with technology-rich intervention: What matters in teaching and

teacher training at scale? (Unpublished doctoral dissertation). Rutgers University, Rutgers, NJ.Empson, S., Greenstein, S., Maldonado, L., & Roschelle, J. (2009). Scaling Up SimCalc: Case studies of enactment.

Manuscript submitted for publication.Fennema, E., Carpenter, T. P., Franke, M. L., Levi, L., Jacobs, V. R., & Empson, S. (1996). A longitudinal study of

learning to use children’s thinking in mathematics instruction. Journal for Research in Mathematics Education, 27(4),403–434.

Floden, R., McCrory, R., Ferrini-Mundy, J., Reckase, M. D., & Senk, S. (2009). Knowledge of algebra for teaching:Validity studies of a new measure. Paper presented at the annual conference of the American Educational ResearchAssociation, San Diego, CA.

Garet, M. S., Birman, B. F., Porter, A. C., Desimone, L., Herman, R., & Yoon, K. S. (1999, October). Designing effectiveprofessional development: Lessons from the Eisenhower Program. Report to the U. S. Department of Education,Office of the Under Secretary on Contract No. EA97001001 to the American Institutes for Research.

Goldhaber, D. (2007). Everyone’s doing it, but what does teacher testing tell us about teacher effectiveness? Journal ofHuman Resources, XLII(4), 765–794.

Good, T., & Grouws, D. (1977). Teaching effects: A process-product study in fourth-grade mathematics classrooms.Journal of Teacher Education, 28(3), 49.

Hanushek, E. A. (1986). The economics of schooling: Production and efficiency in public schools. Journal of EconomicLiterature, 24(3), 1141–1177.

Harbison, R., & Hanushek, E. (1992). Educational performance of the poor: Lessons from rural Northeast Brazil.Washington, DC: World Bank.

Harel, G., & Confrey, J. (1994). The development of multiplicative reasoning in the learning of mathematics. Albany,NY: State University of New York Press.

Hiebert, J., & Carpenter, T. (1992). Learning and teaching with understanding. In D. Grouws (Ed.), Handbook of researchon mathematics research and teaching (pp. 65–100). New York: MacMillan.

Hiebert, J., & Grouws, D. A. (2007). The effects of classroom mathematics teaching on students’ learning. In F. K.Lester (Ed.), Second handbook of research on mathematics teaching and learning (pp. 371–404). Greenwich, CT:Information Age.

Hill, H. C., Blunk, M. L., Charalambous, C. Y., Lewis, J. M., Phelps, G. C., Sleep, L., et al. (2008). Mathematicalknowledge for teaching and the mathematical quality of instruction: An exploratory study. Cognition and Instruction,26(4), 430–511.

Hill, H. C., Rowan, B., & Ball, D. L. (2005). Effects of teachers’ mathematical knowledge for teaching on studentachievement. American Educational Research Journal, 42(2), 371–406.

Hill, H. C., Schilling, S. G., & Ball, D. L. (2004). Developing measures of teachers’ mathematics knowledge for teaching.The Elementary School Journal, 105(1), 11–30.

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


Kaput, J. (1994). Democratizing access to calculus: New routes using old roots. In A. Schoenfeld (Ed.), Mathematicalthinking and problem solving (pp. 77–155). Hillsdale, NJ: Erlbaum.

Kaput, J. (1997). Rethinking calculus: Learning and thinking. The American Mathematical Monthly, 104(8), 731–737.Kaput, J., Hegedus, S., & Lesh, R. (2007). Technology becoming infrastructural in mathematics education. In R. Lesh,

E. Hamilton, & J. Kaput (Eds.), Foundations for the future in mathematics education (pp. 173–192). Mahwah, NJ:Erlbaum.

Kaput, J., & Roschelle, J. (1998). The mathematics of change and variation from a millennial perspective: New content,new context. In C. Hoyles, C. Morgan, & G. Woodhouse (Eds.), Rethinking the mathematics curriculum (pp. 155–170).London: Falmer Press.

Kersting, N., Givvin, K. B., Santagata, R., Sotelo, F., & Stigler, J. W. (2009). Teachers’ analysis of classroom video asa predictor of students’ mathematics learning: Further explorations of a novel measure of teacher knowledge. Paperpresented at the annual meeting of the American Educational Research Association, San Diego, CA.

Leinhardt, G., Zaslavsky, O., & Stein, M. (1990). Functions, graphs, and graphing: Tasks, learning, and teaching. Reviewof Educational Research, 60, 1–64.

Ma, L. (1999). Knowing and teaching elementary mathematics: Teachers’ understanding of fundamental mathematics inChina and the United States: Mahwah, NJ: Erlbaum.

Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. EducationalResearcher, 23(2), 13–23.

Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to evidence-centered design. CRESST TechnicalPaper Series. Los Angeles: Center for Research on Evaluation, Standards, and Student Testing (CRESST).

Mislevy, R. J., & Haertel, G. (2007). Implications of evidence-centered design for educational testing. EducationalMeasurement: Issues and Practice, 25, 6–20.

Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2002). On the structure of educational assessments. Measurement:Interdisciplinary Research and Perspectives, 1, 3–67.

Monk, D. H. (1994). Subject area preparation of secondary mathematics and science teachers and student achievement.Economics of Education Review, 13, 125–125.

Mullens, J., Murnane, R., & Willett, J. (1996). The contribution of training and subject matter knowledge to teaching effec-tiveness: A multilevel analysis of longitudinal evidence from Belize. Comparative Education Review, 40(2),139–157.

National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. Reston, VA:Author.

National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the National MathematicsAdvisory Panel. Washington DC: U.S. Department of Education.

No Child Left Behind Act. (2001). United States Congress.Nye, B., Konstantopoulos, S., & Hedges, L. (2004). How large are teacher effects? Educational Evaluation and Policy

Analysis, 26(3), 237–257.Pierson, J. (2008). The relationship between patterns of classroom discourse and mathematics learning. (Unpublished

doctoral dissertation). University of Texas at Austin.Porter, A. C. (2002). Measuring the content of instruction: Uses in research and practice. Educational Researcher, 31(7),

3–14.Post, T. R., Cramer, K. A., Behr, M., Lesh, R., & Harel, G. (1993). Curriculum implications of research on the learning,

teaching, and assessing of rational number concepts. In T. A. Romberg, E. Fennema, & T. P. Carpenter (Eds.), Rationalnumbers: An integration of research (pp. 327–362). Hillsdale, NJ: Erlbaum.

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.).Newbury Park, CA: Sage.

Roschelle, J., Knudsen, J., & Hegedus, S. (2009). From new technological infrastructures to curricular activity systems:Advanced designs for teaching and learning. In M. J. Jacobson & P. Reimann (Eds.), Designs for learning environmentsof the future: International perspectives from the learning sciences (pp. 233–262). New York: Springer.

Roschelle, J., Shechtman, N., Tatar, D., Hegedus, S., Hopkins, B., Empson, S., Knudsen, J., & Gallagher, L. (in press).Integration of technology, curriculum, and professional development for advancing middle school mathematics: Threelarge-scale studies. American Educational Research Journal.

Roschelle, J., Singleton, C., Sabelli, N., Pea, R., & Bransford, J. (2008). Mathematics worth knowing, resources worthgrowing, and research worth nothing: A response to the National Mathematics Advisory Panel report. EducationalResearcher, 37(9), 610–617.

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


Shechtman, N., Haertel, G., Roschelle, J., & Knudsen, J. (2010). Design and development of the student and teachermathematical content assessment instruments for the Scaling Up SimCalc Project (Scaling Up SimCalc ProjectTechnical Report No. 05). Menlo Park, CA: SRI International.

Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14.Sowder, J., Philipp, R., Armstrong, B., & Schappelle, B. (1998). Middle-grade teachers’ mathematical knowledge and

its relationship to instruction. Albany, NY: State University of New York.Tatar, D., Roschelle, J., Knudsen, J., Shechtman, N., Kaput, J., & Hopkins, B. (2008). Scaling up innovative technology-

based mathematics. Journal of the Learning Sciences, 17(2), 248–286.U. S. Department of Education. (1996). Pursuing excellence: A study of U.S. eight grade mathematics and science

teaching, learning, curriculum, and achievement in an international context. Washington DC: U.S. GovernmentPrinting Office.

Vergnaud, G. (1988). Multiplicative structures. In J. Hiebert & M. Behr (Eds.), Number concepts and operations in themiddle grades (pp. 141–161). Reston, VA: National Council of Teachers of Mathematics.

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


APPENDIX A: SAMPLE MKT ITEMS ORGANIZED BY THE SIMCALC MKTCONCEPTUAL FRAMEWORK

1. Link and translate between precise aspects of functional representations (i.e., story, graph,table, algebra). These translations are important to the Texas eighth-grade standards andare directly emphasized in the SimCalc replacement curriculum for eighth grade.

A teacher wants to illustrate to her class that different representations can represent thesame function. Which of the following are mathematically valid illustrations? (Chooseall that apply)

Correct answer: A

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


2. Evaluate the validity of students’ mathematical conjectures. The SimCalc seventh-gradeunit contains a lesson that focuses on a graph like this, and we have observed students tomake conjectures of this type. Thus, the relevant mathematics is important for a teacherto know.

A student observes the following graph representing 4 successive trial runs of one runner.

The student makes this conjecture, “I see the pattern: the shorter the line, the faster the run.”Under which conditions is this statement true about one-segment straight line graphs ofruns that all start at the origin? (Choose all that apply.)

A. It’s always trueB. It’s true whenever the runs are all the same distance.C. It’s true whenever the 4 runs are increasingly short (shorter and shorter) distances.D. It’s true whenever the 4 runs are increasingly short (shorter and shorter) times.E. It’s true whenever the lines are at different angles.F. None of the above.

Correct answer: B

3. Differentiate between colloquial and mathematical uses of language and evaluate studentstatements for their mathematical precision. In prior SimCalc research, we found thatstudents often have difficulty choosing appropriate mathematical language for the slopeof a line. It is important for a teacher to be able to model and scaffold precise language.

Given the graph below, which statement below is most mathematically precise? (Chooseone answer)

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


A. The slope of the line is horizontal.B. The lines is flat.C. The slope of the line is 0.D. All of the above statements are equally mathematically correct.E. None of the above statement are mathematically precise.

Correct answer: C

4. Interpret common unconventional (in many cases, mathematically correct) forms orrepresentations that students are likely to make as they construct their understanding.Students often transpose the axes that independent and dependent variables conventionallyappear on. As students are asked to label and draw graphs in the curriculum unit, teachersneed to know how to make sense of unconventional student graphs.

A student made this graph of a 50-meter dash. Notice that distance is on the x-axis andtime is on the y-axis.

Which are true statements about the relationship between the line graph and the speed ofthe runner? (Choose all that apply.)

A. The slope of the line is 9/50 or .18, as was the average speed in meters per second ofthe runner during the dash.

B. The slope of the line is 50/9 or 5.56, as was the average speed in meters per secondof the runner during the dash.

C. The slope of the line is 9/50 or .18, and the average speed of the runner was 50/9 or5.56 meters per second.

D. The slope of the line is 50/9 or 5.56, and average speed of the runner was 9/50 or .18meters per second.

E. None of the above

Correct answer: C

5. Generate, choose, and evaluate problems and examples that can illustrate key curricularideas. This content is core to the proportionality focus in the seventh-grade unit. Teachersoften need to construct additional problems spontaneously to help students, and thus needto evaluate whether a problem makes sense for a proportional relationship.

Ms. Kay considers three problems for illustrating the use of ab

= cd

A. Both Sam’s and Fred’s cars travel at the same constant rate. Sam starts at the 10 mmarks. Fred starts at the 20 m mark. When Sam is at 50 m, where is Fred?

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


B. Both Sam’s and Fred’s cars travel at the same constant rate. Sam travels 10 m in2 seconds. Fred travels 20 m. How many seconds does it take Fred to travel that far?

C. Sam’s and Fred’s cars travel the same average rates. Sam travels 100 m in 10 seconds.Fred travels for 14 seconds at an increasing rate. How far does Fred travel?

Which story(ies) are appropriate for solving with ab

= cd

? (Choose all that apply.)

A. Story A.B. Story B.C. Story C.D. None of the above.

Correct answer: B

6. Make connections to important advanced mathematics beyond the unit. SimCalctraditionally extends toward ideas in Calculus, such as the relationship between positionand velocity functions. Because this mathematics was addressed in the seventh-gradeworkshop for teachers, we included items such as this to test for teacher learning ofadvanced mathematics.

Correct answer: C

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


APPENDIX B. SAMPLE CHARACTERISTICS

Sample Sizes By Study and Group

Seventh-Grade Seventh-Grade Eight-GradeYear 1 Experiment Quasi-Experiment Experiment

Group NTeachers NStudents∗ Group NTeachers NStudents

∗ Group NTeachers NStudents∗

Delayed 47 825 Year 1 30 510 Control 23 303Immediate 48 796 Year 2 538 Treatment 33 522Total 95 1,621 Total 30 1,048 Total 56 825

∗Only students for whom we have complete data (both pretest and posttest) are included here.

Teacher Characteristics

Seventh-GradeSeventh-Grade Quasi-Experiment Eighth-Grade

Year 1 Experiment ExperimentDelayed-Treatment Teachers

Variable Delayed Immediate Who Completed Years 1 and 2 Control Treatment

Total count 47 48 30 23 33Teachers by region

Region 1 (Edinburg) 11 8 6 — —Region 9 (Witchita Falls) — — — 4 3Region 10 (Dallas) — — — 4 8Region 11 (Fort Worth) 13 14 8 — —Region 13 (Austin) 13 11 8 10 13Region 17 (Lubbock) — — — 5 6Region 18 (Midland) 10 15 8 0 3

Female (%) 81 77 80 82.6 84.8Years teaching total

Mean 10.5 12.4 10.3 9.6 7.9Range 1–29 1–40 1–27 (+1 in year 2) 0–27 0–31

Years teaching mathMean 9.5 11.0 9.0 9.9 8.2Range 1–29 1–40 1–27 (+1 in year 2) 0–27 1–32

Teacher ethnicity (%)White 70.2 77.1 70.0 87.0 78.8Hispanic 25.5 20.8 23.3 8.7 15.1Asian 4.3 0 6.7 0 0African American 0 2.1 0 4.3 6.0

Master’s degree (%) 17.0 18.8 16.7 26.1∗ 6.0

∗p = .06; significance tests compared groups within study.Note. Within each study, no significant differences existed between groups on any of these variables.

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014


School Characteristics

Seventh-GradeSeventh-Grade Quasi-Experiment Eighth-Grade

Year 1 Experiment ExperimentDelayed-Treatment Teachers

Variable Delayed Immediate Who Completed Years 1 and 2 Control Treatment

Total count of schools 37 36 25 19 23Free/reduced-price lunch (%)

Mean 49 54 53 47 43Range 4–99 1–94 11–99 12–89 0–92

Campus ethnicity (mean%)White 48 49 43 59 57Hispanic 44 45 48 30 35Asian 2 2 2 1 1African American 6 4 6 9 6

Note. Within each study, no significant differences existed between groups on campus enrollment, free/reduced-pricelunch, or campus ethnicity.

Student Characteristics

Seventh-Grade Seventh-Grade Eighth-GradeYear 1 Experiment Quasi-Experiment Experiment

Variable Delayed Immediate Year 1 Year 2 Control Treatment

Total count of students 825 796 510 538 303 522Female (%) 50.6 48.9 52.4 41.9∗∗ 45.1 47.9Individual ethnicity (%)

White 38.7 48.5 39.6 35.4 65.6 50.0Hispanic 54.1 44.3 51.7 55.8 22.7 40.7∗Asian 2.0 1.5 2.8 2.6 1.1 1.3African American 4.7 4.2 5.5 5.3 9.5 6.9

Achievement level (%)Low 26.2 22.5 26.9 27.5 23.4 25.1Medium 42.9 35.9 45.3 41.1 42.4 37.7High 24.2 28.6 25.9 25.5 25.4 26.6

∗∗p < .01; ∗p < .05; significance tests compared groups within study using a three-level HLM model.Note. Data obtained from teacher report of students.

Dow

nloa

ded

by [

Fond

ren

Lib

rary

, Ric

e U

nive

rsity

] a

t 17:

20 2

2 N

ovem

ber

2014

Documents

Investigating Links from Teacher Knowledge, to Classroom Practice, to Student Learning in the Instructional System of the Middle-School Mathematics Classroom