1

Measuring instructional practices in mathematics using a daily log

Carrie W. Lee, Temple A. Walkowiak, and Elizabeth L. Greive North Carolina State University

Paper for presentation at National Council of Teachers of Mathematics 2014 Research Conference

April 2014 New Orleans, LA

This work is funded by the National Science Foundation under Award #1118894. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of NSF. Questions or comments about the work in this paper should be directed to the Principal Investigator and second author of this paper at temple_walkowiak@ncsu.edu.

mailto:temple_walkowiak@ncsu.edu�

2

Abstract

In the current era of accountability with a push to use value-added test scores for evaluation of

programs and teachers, there is a need for valid and reliable measures of mathematics teaching

practices that can be used in a large number of classrooms. In this paper, we present the

Mathematics Instructional Log, an instrument completed by elementary school teachers, that

aims to measure three domains of standards-based mathematics teaching (tasks, discourse, and

representations) that are aligned with the Standards for Mathematical Practice in the Common

Core and with NCTM’s process standards. The purposes of the paper are: (1) to describe the

theoretical framework underpinning the Mathematics Instructional Log; (2) to discuss the

development process; (3) to share the results of an exploratory factor analysis of pilot data; and

(4) to outline implications for future work. Initial evidence of validity and score reliability are

discussed.

3

Introduction

The mathematics achievement of America’s students has remained a topic of concern and

discussion for policy makers and educators for several decades. When television, newspapers,

and magazines publish headlines like “Sluggish Results Seen in Math Scores” (Dillon, 2009) and

“How to Solve Our Problem with Math” (Ramirez, 2008), even the general public engages in the

conversation. This attention to mathematics stems from two sources. First, today’s global

economy has resulted in newly created jobs that require mathematical problem solving and

technological skills (Glenn, 2000). Second, comparisons on international assessments of

industrialized countries indicate that American students have consistently performed in the

bottom third in mathematics (Cooke, Ginsburg, Leinwand, Noell, & Pollock, 2005).

Most recently, the development, adoption, and implementation of the Common Core

State Standards in Mathematics (CCSS-M) (National Governor’s Association Center of Best

Practices, Council of Chief State School Officers, 2010) has drawn even more attention from

researchers, practitioners, and the public as to what is happening inside mathematics classrooms

in the United States. Alongside the new standards, we live in an age with increasing attention to

value-added models for teacher accountability. In addition to this scrutiny of K-12 schools and

teachers, colleges of education have been under fire for their work in preparing the next

generation of teachers with more attention to evaluating and ranking the effectiveness of teacher

preparation programs (Greenberg, McKee, & Wash, 2013). With all of these current challenges,

there is an immediate need for tools that produce data on classroom practices beyond student

achievement outcomes. A careful examination of students' experiences during school

mathematics is certainly warranted, considering this explicit focus in the Standards for

Mathematical Practice in the CCSS-M.

4

The recent adoption and subsequent implementation of the CCSS-M by many states

coupled with the increased need to carefully evaluate teacher preparation programs in response

to scrutiny, result in an urgency to describe the type of mathematics instruction happening in

schools as it relates to the practices outlined in the new standards. In response to the need to

catalogue and quantify teachers’ mathematics instruction as a part of a program evaluation, the

Mathematics Instructional Log was developed. At its core, it describes the frequency and type of

mathematics teaching practices present in elementary mathematics lessons. Although in its early

stages of development, the purposes of this paper are to: (1) outline the instrument’s theoretical

framework; (2) describe the development process to date; (3) share the results of the pilot study;

and (4) discuss implications and next steps for future work.

Significance and Context of the Work

The development of the Mathematics Instructional Log occurred in the context of an

ongoing five-year research study, Project ATOMS (Accomplished Elementary Teachers of

Mathematics and Science), in which we are evaluating the effectiveness of a STEM-focused

elementary teacher preparation program. In this study, we are examining the development of our

teacher candidates while undergraduate students into their first two years of teaching,

particularly in relation to STEM content. While the study has many facets, we are most

interested in their knowledge, beliefs, and teaching practices and the relationships among these

constructs.

Teachers’ instructional practices remain a central focus in this age of accountability,

particularly among teacher educators and school administrators. The recent shift to a focus on

accountability has created a need for valid and reliable evaluative measures of teaching practice.

While Race to the Top pushes for evaluation of program effectiveness using student test score

5

gains (U.S. Department of Education, 2009), the applicability of student test score gains for the

purposes of program evaluation is limited by tested subjects and grade levels, growth model

selection, and characteristics of teachers and schools (Henry, Kershaw, Zulli, & Smith, 2012).

The field of education is challenged by a lack of promising evaluative measures that are focused

on unique program features in order to measure program effectiveness. Due to these limitations,

evaluators of education programs are often forced to develop instruments and conduct validation

activities as part of routine evaluation efforts; this is the very focus of the work described in this

paper about the Mathematics Instructional Log.

In the past, there have been two commonly used tools for gathering data on instructional

practices, observations (e.g., Hiebert et al., 2005) and surveys (e.g., Early Childhood

Longitudinal Study). Classroom observations have often been called the “gold standard”, but the

cost of implementing observations is expensive. Additionally, Rowan and Correnti (2009) found

large variability in practices across days, indicating a large number of observations are needed to

reliably discriminate among teachers. Furthermore, extensive training is often needed to ensure

inter-rater reliability. Surveys, while cost-effective, have been criticized for the accuracy of data

collected (e.g., Mayer, 1999). Most surveys are administered as annual surveys and therefore

require teachers to retrospectively answer questions about their instruction over long periods of

time. This form of data collection introduces issues of memory error and estimation strategies

(Rowan, Jacob, & Correnti, 2009).

Another tool used less often is an instructional log (Rowan, Harrison, & Hayes, 2004:

Rowan, Camburn, & Correnti, 2004; Camburn & Barnes, 2004), a daily questionnaire in which

teachers “log” about instruction. While instructional logs have limitations of their own, this

approach has been explored as a solution to the shortcomings of observational and survey

6

measures. Instructional logs allow researchers to collect data across a large number of teachers,

and in turn, examine patterns across thousands of days. As a result, logs address the limitations

of observational measures by allowing teaching variation to be documented through the increase

in data collection points. Additionally, instructional logs are less expensive than the cost of

conducting observations. In regard to concerns with survey measures, when teachers complete

the log on the day of instruction, issues of memory lapse are addressed. It is also common for

annual surveys to ask broader questions due to the retrospective nature of the task. With daily

instructional logs, more specific questions can be asked, allowing for more information to be

collected about daily instruction.

The use of a mathematics instructional log has proven to be a valid measure of

mathematics teaching practices (Rowan, Harrison, & Hayes, 2004) in work done by researchers

on the Study of Instructional Improvement (SII). In their work, they collected data on three

dimensions of teaching: whether or not a teacher used direct teaching, the pacing of content

coverage, and the nature of students’ mathematical work. They made a decision to only collect

this information around three focus topics: number concepts; operations; and patterns, functions,

or algebra. While their work offered compelling results and greatly informed the work outlined

in this paper, the features of our program along with the goals of our research study prompted us

to develop the Mathematics Instructional Log to address the theoretical framework driving our

work.

Theoretical Framework

The theoretical framework that guided the development of our instructional log focuses

on three domains of mathematics instruction: tasks, discourse, and representations (Berry,

Rimm-Kaufman, Ottmar, Walkowiak, & Merritt, 2010; Walkowiak, Berry, Rimm-Kaufman,

7

Meyer, & Ottmar, 2014). These domains are the overarching focus within the process standards

outlined by NCTM’s Principles and Standards for School Mathematics (2000) and within the

Standards for Mathematical Practice in the CCSS-M. For example, two of the mathematical

practices, “model with mathematics” and “use appropriate tools strategically,” align well with

representations, also one of the process standards. The instructional log captures information

about the opportunities that are provided within a lesson in regard to tasks, discourse, and

representations. The log is not able to address the same level of depth that observational

measures are able to capture, but the log is able to capture frequency and presence of certain

activities within these domains.

Tasks. First, the tasks in which students engage during mathematics instruction matter.

One characteristic to consider is a task's level of cognitive demand, based upon the type of

thinking required of students. The cognitive demand framework (Stein, Smith, Henningsen, &

Silver, 2009) organizes tasks into four categories based on the amount of rigor and use of

conceptual understanding. The lower level categories, memorization and procedures without

connections, focus more on completing tasks based on a set of memorized steps without tying the

process to mathematical meaning. The higher level categories, procedures with connections and

doing mathematics, involve tasks that require explanation or use of conceptual understanding

within a mathematical process. While students may engage in tasks at all levels, analysis of the

frequency of tasks within each category allows conclusions to be made about students’

opportunities to engage with higher level tasks. Students who authentically engage in higher

level tasks develop reasoning skills that are emphasized in the Standards for Mathematical

Practice in the CCSS-M. While the mathematics log cannot provide information about the

authenticity of the tasks in which students engage, the log does provide information about the

8

frequency of tasks that are made available to the students. The log lists 44 items that describe

different tasks or activities that the daily lesson may include. By indicating which activities

occurred during the lesson, information can be gleaned about the nature of the cognitive demand

provided to the students.

Discourse. Second, researchers in mathematics education (Lampert & Blunk, 1998;

Hufferd-Ackles, Fuson, & Sherin, 2004) emphasize that discourse is an important component of

school mathematics because it is a central part of what and how students learn. Student-to-

student and student-to-teacher discourse during which students explain and justify their

reasoning, is a critical component of standards-based practices that have been shown to result in

better learning outcomes (Stein, 2007; Truxaw & DeFranco, 2007; Truxaw, Gorgievski &

DeFranco, 2008). Communication is also a foundational tenet of the mathematical practices

outlined by both NCTM and the CCSS-M. A teacher's role is a primary support for student

talk. The mathematics log includes several items that address discourse by indicating if

communication of mathematical ideas occurred among the students. The log provides a picture

of the frequency of student talk during a lesson. Although, the log cannot provide a detailed

account of what strategies are used or the depth of implementation, it can show that discourse

opportunities were included in the lesson.

Representations. Third, Lesh, Post, and Behr (1987) outline five representations students

and teachers can use for mathematical concepts: pictures, written symbols, oral language, real-

world situations, and manipulative models. Research has shown the importance of using

multiple representations in mathematics instruction to help students create understanding (e.g.,

Lehrer & Schauble, 2002), but the use of multiple representations should include making explicit

connections among these representations (Duval, 2006). Within the mathematics log there are

9

items when the teacher responds with representations he/she and the students used during

instruction. The distinction between the teacher’s and students’ use is important because it

provides information about the type of representations students are using themselves in their

mathematics learning and the presence of multiple forms. Also, log items ask whether the teacher

and/or the students made explicit connections among the representations.

Development of the Log

The Mathematics Instructional Log was developed to measure mathematics instructional

practices that occur in elementary classrooms, specifically instructional practices as outlined in

the above theoretical framework. The development of the Mathematics Instructional Log was

grounded in methods of educational testing and measurement (Kubiszyn & Borich, 2010).

Starting in August 2012 and continuing to date, the log development process has included five

overlapping and iterative stages as outlined in Table 1.

Table 1: Development Process of the Mathematics Instructional Log

Stage Focus

1 Development of purpose and theoretical framework

2 Development of items and scales

3 Development of training for teachers and implementation of the first pilot

4 Revisions of initial log and implementation of the second pilot with revised log

5 Revisions of second (revised) log; Ongoing and continued validation efforts

The development process is ongoing and focuses on examining the validity and score

reliability of the instructional log, a critical piece of measurement development. This work uses

10

the integrated conception of validity proposed by Messick (1993) to develop and begin to

validate the instructional log. In the integrated conception, construct validity is not easily

separated from other types of validity and is a unitary concept.

In Stage 1, the development of the theoretical framework was heavily grounded in the

practices or processes specified in the CCSS-M (2010) and NCTM’s process standards (2000),

focusing specifically on standards-based teaching practices which are emphasized in the

elementary education program that the instrument was designed to evaluate. Also, questions

about mathematical content were designed to incorporate the content standards of the CCSS-M

to align the log to current reform and to allow for collection of information on content topics

taught across time points. As a part of this stage, existing mathematics instructional logs were

analyzed, including the log created by the Study of Instructional Improvement (Rowan, Harrison,

& Hayes, 2004) and the log developed within the Mosaic II Study (Le, Stecher, Lockwood,

Hamilton, & Robyn, 2006).

Stage 2 focused on item and scale development. Existing items from the SII and Mosaic

II logs were analyzed for convergence with specific research goals for the development of the

Mathematics Instructional Log, and new items were drafted to align with the variety of

instructional practices used in mathematics. Both content and practice items were then reviewed

by content experts, and modifications were made accordingly. In addition to expert reviewers,

five elementary teachers participated in cognitive interviews to isolate items that were

ambiguous or biased. As teachers reflected on their most recent mathematics lesson, they were

asked both to identify any items that were confusing and to share how their typical math

instruction was captured by the log. Specifically, they were asked if there were math practices

that they used that were not listed within the log.

11

An initial draft of the log was piloted with 57 teachers in two local school districts in

January 2013 during Stage 3 of the development process. All teachers participated in a 90-

minute face-to-face training. The purpose of the training was to ensure common understanding

among participants of the terms and scales used in the mathematics log and a science log (not

detailed in this paper). The 57 teachers collectively logged 585 days of mathematics instruction.

They completed the log electronically via Qualtrics, a survey software, as soon as possible after

instruction occurred across a timeframe of fifteen school days.

During this first pilot, we also collected qualitative feedback from participants to provide

further clarification to items. Based on feedback from participants during the first pilot, several

items were split to provide further clarity. For example, an item initially listed as “Work on

problem(s) that have multiple answers or solution methods” was separated into “Work on

problems that have multiple answers” and “Work on problems that have multiple solution

methods.” Also, several items were further elaborated to include language concerning discourse

to provide a better picture of what the task included. For example, the item “Prove that a solution

is valid or that a method works for all similar cases” was transformed into two items, “Prove

orally that a solution is valid or that a method works for all similar cases” and “Prove through

written work that a solution is valid or that a method works for all similar cases” to discern if

students engaged in student talk during the task.

The response scale for the majority of items was also revised such that each response

choice was defined clearly. The scale was changed to measure the time that the students

engaged in behaviors rather than the teacher’s perceived emphasis on each behavior, which had

been the focus of the first draft of the log. The scale on the second draft was modified to a four-

point scale for the student behaviors listed after the following question stem, “During today's

12

mathematics instruction, how much time did students.” Teachers were required to each of the

items with one of the following choices:

Not today: This behavior was not done during today’s instruction;

Little: This behavior made up a relatively small part of the instruction;

Moderate: This behavior made up a large portion, but NOT the majority of instruction;

Considerable: This behavior made up the majority of today’s mathematics instruction.

Stage 4 included a second pilot test of the instrument with 54 elementary teachers.

Because the teachers were not located in a central location and we wanted to pilot virtual

trainings, the 90-minute training was conducted virtually using Blackboard Collaborate, which

allowed for teachers to ask questions, respond to polls, and hear the presenter while viewing the

PowerPoint slides. After the training, teachers logged about their mathematics instruction for 15

days. The data from this second pilot were used to conduct an exploratory factor analysis;

results are outlined in the next section of this paper. Currently, as part of Stage 5, 74 second-year

teachers are logging about their mathematics instruction for a total of 45 logged days across the

school year. Validation efforts are ongoing and are discussed later in this paper.

Exploratory Factor Analysis: Results from the Second Pilot Study

Data from the second pilot with 57 teachers was analyzed using an exploratory factor

analysis (EFA) with each day of instruction as a data entry (n=750). The number of mathematics

instructional days logged ranged from 2 to 16 days per teacher, with a mean of 13 days per

teacher. The revised Mathematics Instructional Log included 11 items with a total of 53 sub-

items, all of which were completed by teachers during each logging session. Sub-items were

scored on a Likert scale. Items were arranged in four main sections displayed in Table 2:

content, time spent on mathematics, use of representations, and student activities/behaviors

13

during mathematics instruction.

Table 2: Overview of the Mathematics Instructional Log by Section

Section Log Section Description

Number of Items Examples of Sub-Items

Response Scale

1 Mathematical Content

1 item with 5 sub-items

-Number and Operations:Fractions -Measurement and Data

1=Not today 2=Secondary focus 3=Primary focus

2 Time in minutes spent teaching mathematics

1 item N/A Dropdown in 5 minute intervals

3 Use of representations

2 items with 6 sub-items each

-Number or symbols -Concrete materials -Pictures or diagrams

Dichotomously scored with a checkbox for each item

4 Student activities and behaviors

7 items (all the same stem) with 42 sub-items

-Pose questions to the teacher about the mathematics -Work on today’s mathematics homework

1=Not today 2=Little 3=Moderate 4=Considerable

The exploratory factor analysis included the 2 items in Section 3 and the 42 sub-items in

Section 3, for a total of 44 items in the analysis. They were chosen because they are designed to

measure the three domains of the theoretical framework: tasks, discourse, and representations.

The two items in the Section 3 on representations included one item which asked “What did the

students use to work on mathematics today?” with the following options to check if present:

numbers or symbols; concrete materials; real-life situations or word problems; pictures or

diagrams; tables or charts; and “the students made explicit links between two or more of these

representations. The second item in Section 3 was a parallel item about the representations the

teacher used during the mathematics lesson. These items were rescored on a four-point scale

based upon what was selected with more weight given to the last choice. A four-point scale was

14

used so that these two items would be on a four-point scale like the sub-items in Section 4. The

sub-items from Section 4 include the same item stem mentioned earlier, “During today’s

mathematics instruction, how much time did the students.” The items in Sections 1 and 2,

mathematics content and time spent teaching mathematics, were primarily asked for descriptive

purposes and will be used in future analyses.

Exploratory factor analysis using Principal Axis Factoring in SPSS with oblique rotation

(i.e., Promax) was conducted; oblique factor solutions of 3-7 factors were considered.

Cronbach's alphas were calculated for each factor subscale. We decided on the six-factor Promax

structure, which explained 39.45% of the variance, because the factor loadings suggested clear

cut-off points, the items loaded in a way that most clearly matched our theoretical framework,

and the scree plot showed leveling at seven factors. Table 3 outlines the six factors, alphas for

the subscales, the range of factor loadings, the percent of variance explained, and sample items.

Three items were dropped because they did not clearly load on a factor (“Connect today's math

topic to another math topic,” “Review or practice math facts that they have memorized”,

“Perform tasks focused on math procedures”).

Table 3: Results of Exploratory Factor Analysis, Pilot #2

Factor # of Items

Alpha Range of Factor Loading

Percent of Variance Explained

Sample Items

Discourse 9 .864 .764-.461 17.96 Pose questions to other students about the mathematics.

Talk about similarities and differences among various solution strategies.

Lower Level Tasks

10 .762 .909-.331 7.29 Listen to me explain the steps to a procedure.

Read from a textbook to learn information.

15

Problem Solving 7 .782 .715-.307 5.51 Demonstrate different ways to solve a problem.

Write explanations of mathematical ideas, solutions, or methods.

Connections 5 .764 .891-.341 3.70 Connect today's math topic to a “real world” idea.

Read from a picture book.

Prior Knowledge

5 .442 .777-.350 2.68 Review mathematics content previously covered.

Participate in an activity designed to activate prior knowledge.

Representations 5 .607 .818-.164 2.31 Use hands-on tools to explore mathematical ideas or to solve problems.

Use pictures or diagrams to represent mathematical concepts.

Although it is not in the limits of this paper to detail the analysis process of the initial

pilot data, an EFA of the same form was conducted that resulted in similar factor loadings to the

larger second pilot. It is important to note these similar loadings resulted despite the fact that

revisions were made between the first and second pilot implementations of the log. The initial

pilot EFA resulted in a five-factor structure with factors closely aligned with the second pilot

EFA; the five factors were identified as cognitive demand, connections and applications,

problem solving, representations, and discourse. In the second pilot, the latter three factors

emerged and were given the same identifying label. The factor of cognitive demand emerged

again, but it was renamed accordingly as lower level tasks due to the nature of the items that

loaded within the factor. The factor, “connections and applications” emerged similarly, but in

this second pilot, the label “connections” was a better fit for describing the set of items. A sixth

factor, “prior knowledge”, which was not present in the first EFA, was a factor resulting from the

16

second pilot EFA.

In addition to support from the initial pilot, it is important to note how the six factors

from the second pilot connect to the theoretical framework. While discourse and representations

factors match corresponding constructs of the theoretical framework, the “tasks” domain

emerges as four factors: lower level tasks, problem solving, connections, and prior knowledge.

Items loadings to the lower level tasks included “Listen to me present the definition for a term,”

“Listen to me explain the steps to a procedure,” and “Orally answer recall questions.” The nature

of these activities are more centered on the direct actions of the teacher and likely do not require

students to make conceptual connections to their work. When analyzed using the task analysis

guide developed by Stein, Smith, Henningsen, and Silver (2009), the nature of the items within

this factor are categorized as memorization tasks or procedures without connections tasks. The

practices and activities fall into these categories because at the surface level, they do not require

students to engage in complex thinking or attend to the conceptual ideas underlying the

procedures. Although one cannot be certain how the teacher implements the task, if other items

from the log are not selected to indicate other practices, tasks in this factor are best characterized

as lower level tasks.

Furthermore, items within the named factor problem solving entailed higher-level skills

such as proving and demonstrating various solution strategies. Examples of items that loaded

within this factor included “Prove through written work that a solution is valid or that a method

works for all similar cases” and “Demonstrate different ways to solve a problem.” These items

describe mathematical activities that require the students to move past using procedures to

engage in tasks that likely demonstrate conceptual understanding of the mathematics. Research

shows that students that engage in tasks that require higher level cognitive process perform better

17

on problem solving and reasoning measures (Henningsen & Stein, 1997). Furthermore, the level

of cognitive demand is sustained by requiring students to justify or explain their thinking which

are actions that are captured by log items in this factor grouping.

Additionally, connections and prior knowledge are distinct components of tasks

captured by the log. The items that loaded within the factor of connections explicitly stated that

connections were made to another subject, other math concepts, or real-world situations. Also,

activities with picture books loaded within this factor, which is understandable due to the real-

world setting of most picture books designed to integrate mathematical concepts. The emergence

of the connections factor sheds light on the unique characteristics of activities that are designed

to help students link mathematical processes to the world around them. Henry Kepner, NCTM

President speaks to the importance of connections within mathematics in his statement, “When

students connect mathematical ideas, their understanding becomes deeper and more lasting, and

learners come to view mathematics as a coherent whole—connected with other subjects and their

own interests and experiences” (NCTM Summing Up, 2009). Prior knowledge was a weaker

factor with an alpha level of only .422. The items that loaded within this factor included

activities that review previous learning and activities that activate prior knowledge. These items

do not directly relate to one another, and therefore, the construct they illustrate within the factor

does not seem as clear as the previous ones described.

Also, it appears it may be more difficult to capture representations with only four items

and some weaker loadings. Some of the items that might be considered a measure of

representations loaded under a different factor. For example, the item “Talk about similarities

and differences among representations” loaded within the discourse factor, yet the item still

provides information concerning the use of representations within the lesson. This speaks to the

18

complexity of the constructs and encourages the research team to consider dual loadings.

Although the total variance explained was 39.45%, the factors show promise for

measuring features of the intended domains of tasks, discourse, and representations. The results

are limited because at this stage in the log development, the analysis did not account for the

nested structure of the data, with logging days situated within individual teachers. Additionally,

the trainings for this second pilot were conducted virtually, which proved to be a challenge to

make sure that participating teachers were attentive during the training.

Attention to evidence of validity and score reliability has been addressed throughout the

development process (and continued collection of evidence is ongoing). First, the cognitive

interviews and feedback from pilot participants examined the face validity of the measure by

asking participants how they interpreted the items, suggesting which items were vulnerable to

inconsistent interpretation. Second, experts reviewed the log's content and agreed the items

appear to measure the constructs in the theoretical framework. Third, construct validity has been

examined through the factor analysis; the three domains of the theoretical framework are

represented in the factor structure. Finally, there is evidence of score reliability with the

Cronbach's alphas for the factor subscales (we recognize the weaker internal consistency of the

“prior knowledge” subscale indicating further data collection and analyses are needed). Our

ongoing and future collection of validity evidence is outlined in the next and final section of the

paper.

Implications and Future Directions

Although in its early stages of development, we believe that describing the development

of our Mathematics Instructional Log can help other researchers interested in ways to measure

teaching practices. This work addresses the need for valid quantitative measures that can be used

19

reliably in order to investigate the teaching practices used in large number of classrooms. Based

on feedback from the participating teachers in the pilots, completing the Mathematics

Instructional Log across a number of instructional days seems to also contribute to a more

reflective and analytic approach towards mathematics teaching.

Future steps in the development of the Mathematics Instructional Log include conducting

in-depth cognitive interviews, nested exploratory (EFA) and confirmatory (CFA) factor analysis,

and item response analysis. In-depth cognitive interviews will provide information about how

teachers are thinking about the items as they respond. Taking into account the nested structure

of the data, the team is currently conducting an EFA and will follow up with a nested CFA to

assess the quality of the item measurement scales or factors. Additionally, the team plans to

apply item response theory to understand the extent to which the factor scales fit the data.

We acknowledge several limitations in using an instructional log. Logs may be prone to

measurement error due to teachers’ self-reports of their own instruction. In addition to

conducting a sample cognitive interviews, we are video recording the teachers’ lessons in order

to understand the extent to which the log is accurately representing a teacher’s mathematics

instruction. Also, all 74 second-year teachers currently completing the log are required to submit

three video recorded mathematics lessons. These videos will be scored by trained raters and

compared to the teacher’s log report. The results will be analyzed as a source of criterion

validity evidence. Another way to address the limitation of measurement error caused by teacher

self-report is to provide training on how to objectively use the log. Training provides common

definitions and explanation of logging processes. Continued support and training throughout the

logging time frame helps promote accuracy and participation. These steps were taken in the pilot

and current implementations of the Mathematics Instructional Log.

20

Finally, this instructional log is meant to measure instruction at a gross level (Rowan,

Jacob, & Correnti, 2009). The log is not designed to capture fine-grained nuances of

instructional practices and does not account for the quality of those practices. Further

understanding as to whether this is an issue will be reached through an analysis of the collected

videos from participants. The instructional log does not differ across grades and was designed

with the intent to use it in elementary (K-5) classrooms. We are uncertain about the applicability

at all grade levels at this point. In our ongoing work, we will continue to investigate these and

other issues related to using an instructional log to measure teaching practices in mathematics.

The Mathematics Instructional Log was developed to address the need for valid

quantitative measures that can be used reliably to examine instructional practices in a large

number of classrooms. With the current need for understanding teaching practices in the age of

the CCSS-M and the need for program evaluation instruments, the work outlined in this paper iis

timely. The Mathematics Instructional Log shows promise as one measure to address these

needs. Measures like the log can add to the existing body of research on mathematics teaching

and offer meaningful implications for teachers, teacher educators, and researchers regarding

mathematics instruction. The work presented in this paper and future analyses will provide

understanding as to the extent to which the Mathematics Instructional Log is a valid and reliable

measure of mathematics teaching practices.

21

References Berry, III, R. Q., Rimm-Kaufman, S. E., Ottmar, E. M., Walkowiak, T. A., & Merritt, E. (2010).

The Mathematics Scan (M-Scan): A Measure of Mathematics Instructional Quality. Unpublished measure, University of Virginia.

Dillon, S. (2009, 2009, October 14). Sluggish results seen in math scores. The New York Times,

pp. Retrieved November 2, 2009 from http://www.nytimes.com. Duval, R. (2006). A cognitive analysis of problems of comprehension in a learning of

mathematics. Educational studies in mathematics, 61(1-2), 103-131 Cooke, G., Ginsburg, A., Leinwand, S., Noell, J., & Pollock, E. (2005). Reassessing US

international mathematics performance: New findings from the 2003 TIMSS and PISA. Washington, DC: American Institutes for Research.

Glenn, J. (2000). Before it's too late: A report to the nation from the national commission on

mathematics and science teaching for the 21st century. Jessup, MD: Education Publications Center.

Greenberg, J., McKee, A., & Wash, K. (2013). Teacher prep review: A review of the nation’s

teacher preparation programs. Washington, DC: National Center on Teacher Quality. Henningsen, M., & Stein, M. K. (1997). Mathematical tasks and student cognition: Classroom-

based factors that support and inhibit high-level mathematical thinking and reasoning,” Journal for Research in Mathematics Education, 28(5), 524–549.

Henry, G. T., Kershaw, D. C., Zulli, R. A., & Smith A. A. (2012). Incorporating teacher

effectiveness into teacher preparation program evaluation. Journal of Teacher Education, 63(5), 335-355.

Hufferd-Ackles, K., Fuson, K.C., & Sherin, M.G. (2004). Describing levels and components of a

math-talk learning community. Journal for Research in Mathematics Education, 35(2), 81-116.

Kubiszyn, T., & Borich, G. (2010). Educational Testing & Measurement: Classroom Application

and Practice (9th ed.). Hokoken, NJ: John Wiley & Sons, Inc. Lampert, M. & Blunk, M. L. (Eds.). (1998). Talking mathematics in school: Studies of teaching

and learning. New York: Cambridge University Press. Le, V. N., Stecher, B. M., Lockwood, J. R., Hamilton, L. S., Robyn, A., Williams, V. L., Ryan,

G. W., Kerr, K. A., Martinez, J. F., & Klein, S. P. (2006). Improving mathematics and science education: A longitudinal investigation of the relationship between reform-oriented instruction and student achievement. Santa Monica, CA: Rand Corporation.

http://www.nytimes.com/�

22

Lehrer, R., & Schauble, L. (2002). Symbolic communication in mathematics and science: Co-constituting inscription and thought. In E. Amsel, & J. P. Byrnes (Eds.), Language, literacy, and cognitive development: The consequences of symbolic communication (pp. 167-192). Mahwah, NJ: Erlbaum.

Lesh, R. A., Post, T. R., & Behr, M. J. (1987). Representations and translations among

representations in mathematics learning and problem solving. In C. Janvier (Ed.), Problems of representation in the teaching and learning of mathematics (pp. 33 - 40). Hillsdale, NJ: Erlbaum.

Mayer, D. P. (1999). Measuring instructional practice: Can policymakers trust surveydata?

Educational Evaluation and Policy Analysis, 21(1), 29–46. Messick, S. (1993). Validity. In R. L. Linn (Ed.), Educational measurement (3rd edition).

Phoenix, AZ: Oryx Press. National Council of Teachers of Mathematics (2000). Principles and standards for school

mathematics. Reston, VA: Author. National Governors Association Center for Best Practices, Council of Chief State School

Officers (2010). Common Core State Standards in Mathematics. Washington, DC: Author.

Ramirez, E. (2008, 2008, December 4). How to solve our problems in math. U.S. News and

World Report, pp. Retrieved November 2, 2009 from http://www.usnews.com. Riordan, J. E., & Noyce, P. E. (2001). The impact of two standards-based mathematics curricula

on student achievement in Massachusetts. Journal for Research in Mathematics Education, 32(4), 368-398.

Rowan, B., Camburn, E., & Correnti, R. (2004). Using teacher logs to measure the enacted

curriculum: A study of literacy teaching in 3rd grade classrooms. The Elementary School Journal, 105(1), 75–102.

Rowan, B., & Correnti, R. (2009). Studying reading instruction with teacher logs: Lessons from

a study of instructional improvement. Educational Researcher, 38(2), 120-131. Rowan, B., Harrison, D. M., & Hayes, A. (2004). Using instructional logs to study elementary

school mathematics: A close look at curriculum and teaching in the early grades. The Elementary School Journal, 105(1), 103-127.

Rowan, B., Jacob, R., & Correnti, R. (2009). Using instructional logs to identify quality in

educational settings. New Directions for Youth Development, 121, 13-31. Stein, C. C. (2007). Let’s talk. Promoting mathematical discourse in the classroom. Mathematics

Teacher, 101 (4), 285-289.

http://www.usnews.com/�

23

Stein, M. K., Smith, M. S., Henningsen, M. A., & Silver, E. A. (2009). Implementing standards-

based mathematics instruction (2nd ed.). New York: Teachers College Press. Truxaw, M. P. & DeFranco, T. C. (2007). Lessons from Mr. Larson. An inductive model of

teaching for orchestrating discourse. Mathematics Teacher, 101(4), 268-272. Truxaw, M. P., Gorgievski, N., & DeFranco, T. C. (2008). Measuring K-8 teachers’ perceptions

of discourse use in their mathematics classes. School Science and Mathematics, 108(2), 58-68.

U.S. Department of Education. (2009). Race to the Top executive summary. Retrieved from

http://www2.ed.gov/programs/racetothetop/executive-summary.pd Walkowiak, T. A., Berry, R. Q., Meyer, J. P., Rimm-Kaufman, S. E., & McCracken, E. R.

(2014). Introducing an observational measure of standards-based mathematics teaching practices: Evidence of validity and score reliability. Educational Studies in Mathematics, 85(1), 109-128.