Milwaukee)Mathematics)Partnership) PhaseII · PDF fileMilwaukee)Mathematics)Partnership) ... This section includes a summary of this year’s work, with additional data and results

Milwaukee Mathematics Partnership Phase II

Sharing in Leadership for Student Success

Annual Report 2012

Dr. Kevin McLeod

Department of Mathematical Sciences University of Wisconsin-Milwaukee

Dr. DeAnn Huinker

Center for Mathematics and Science Education Research University of Wisconsin-Milwaukee

Dr. Kimberly Farley

Associate Dean, Division of Liberal Arts and Sciences Milwaukee Area Technical College

Sharonda Harris

Mathematics Curriculum Specialist Milwaukee Public Schools

Mr. Henry Kranendonk

Center for Mathematics and Science Education Research University of Wisconsin-Milwaukee

This material is based upon work supported by the National Science Foundation under Grant No. 0831615. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation (NSF).

Milwaukee Mathematics Partnership Phase II Year 4 Annual Report 2012

1

Table of Contents

Project Background and Overview ................................................................................................ 2 MMP Phase II Research Areas ...................................................................................................... 3

Summary of Project Activities and Findings 2012 ........................................................................ 5 Research Area 1. Math Teacher Leader Models and School Impact ................................. 5 Research Area 2. Transition to College Mathematics ...................................................... 8

Research Area 3. Mathematical Knowledge for Teaching (MKT) ................................. 18 Appendices ................................................................................................................................... 31

Appendix A. Phase II Research Questions.............................. .................................. ..................32 Appendix B. Research Area 1 HLM Analysis ....................................................... ..................... 33

Appendix C. MMP Online Survey Questions and Constructed Variables ............ ..................... 39 Appendix D. Research Area 2 MATC Placement Data ......................................... ..................... 43

Appendix E. Research Area 3 MKT Analysis ....................................................... ..................... 63 Appendix F. References ......................................................................................... ..................... 85


2

PROJECT BACKGROUND AND OVERVIEW

Milwaukee Public Schools was the 30th largest district in the nation with enrollment for the 2011-2012 school year at 80,098 across 175 schools. The racial profile was 85.6% non-white. Enrollment percentages were American Indian (0.1%), African American (56.2%), Hispanic (23.5%), Asian (5.2%), and White (14.4%). Of these, 80.2% were low income, 19.8% had special education needs, and 10.0% had limited English proficiency. MPS was a District Identified for Improvement under NCLB.

The Milwaukee Mathematics Partnership (MMP) began in 2003, with the aim of improving mathematics achievement for all students in the Milwaukee Public Schools (MPS), and increasing students’ success in transitioning to college mathematics. The University of Wisconsin-Milwaukee (UWM), Milwaukee Public Schools (MPS), and Milwaukee Area Technical College (MATC) have shared in the leadership for student success as core partners to this unique collaboration among a large urban district, a four-year urban university, and a two-year technical college. The partners have remained steadfast and focused on their vision for challenging mathematics. Four goals have driven the work of the MMP since its inception.

Goal 1. Comprehensive Mathematics Framework: Implement and utilize the Comprehensive Mathematics Framework to lead a collective vision of deep learning and quality teaching of challenging mathematics across the Milwaukee Partnership.

Goal 2. Distributed Leadership: Institute a distributed mathematics leadership model that engages all partners and is centered on school-based professional learning communities.

Goal 3. Teacher Learning Continuum: Build and sustain the capacity of teachers, from initial preparation through induction and professional growth, to understand mathematics deeply and use that knowledge to improve student learning.

Goal 4. Student Learning Continuum: Ensure all students, PK-16, have access to, are prepared and supported for, and succeed in challenging mathematics.

Figure 1. The Comprehensive Mathematics Framework.


3

The Comprehensive Mathematics Framework (shown in Figure 1) encapsulated our vision of challenging mathematics. Our goal of distributed leadership led us to define the positions of school-based Mathematics Teacher Leaders (MTL) and district Mathematics Teaching Specialists (MTS). We approached the teacher learning continuum through many activities from revising courses for our teacher preparation program and offering university coursework and regular professional learning opportunities to MTLs and other district teachers for continued professional development. In terms of the student learning continuum, we worked with MPS teachers to ensure implementation of high-quality mathematics curricula at all grade levels, K-12. At the high school level, we developed tools to predict students’ results on college and university placement tests, and to guide subsequent instruction with the aim of improving placement.

The MMP has been able to continue its core strategies of teacher leadership and school learning teams through the 5-year strategic plan adopted by the Milwaukee Board of School Directors (MPS, 2007), and through fiscal support from the State of Wisconsin, with a $10 million budget line from the Office of the Governor in the FY09 budget (and a similar amount in FY10) for the MPS mathematics efforts, specifically to expand the role of the math teacher leaders (Borsuk, 2007). This budget line was transferred to the Wisconsin Department of Public Instruction for FY11, but was eliminated in subsequent state budgets.

Together with additional reductions in state funding, this elimination required the district to re-think its MTL model. Starting in fall 2011, the school-based MTLs were replaced with approximately 55 MTLs, each responsible for 3-4 schools. The transition resulted in new challenges for the MTLs, as they tried to find ways to work effectively with schools which they may have visited only one day a week. In fall 2012, there were yet further changes in the model. The first change was a renaming of the positions to provide consistency with similar positions for literacy. The Math Teacher Leaders (MTL) now have the position title of "Academic Coach–Mathematics" and the Math Teaching Specialists (MTS) now have the position title of "Mathematics Leader." (For consistency with previous reports, we will continue to use the original position titles of MTL and MTS.) The second change was the identification of a selected set of approximately 40 “focus schools” and the assignment of an MTL to each of those schools, under the direction of the district MTSs and Mathematics Curriculum Specialist. While some additional schools were served by MTLs funded from other sources, the current arrangement left many MPS schools without the support of a school-based MTL.

Important personnel changes also occurred within MPS district leadership. The Chief Academic Officer, Dr. Heidi Ramirez, resigned in July 2012. The district then hired Darienne Driver in the newly created position of Chief Innovation Officer and named Tina Flood as the new Chief Academic Officer. MMP Phase II Research Areas

For Phase II of the MMP, we chose to focus on three specific research areas arising from our earlier work. These research areas include an examination of the impact of Math Teacher Leader models on school communities, the transition of high school students to college mathematics, and the development of mathematical knowledge for teaching and its relationship to teacher leader practice, school communities, and student learning.


4

Research Area 1. Math Teacher Leader Models and School Impact

The MMP first initiated a no-release model of teacher leaders, then transitioned to a partial-release (80%) model for about 100 leaders with $5 million in district funds and $10 million from the State of Wisconsin. We have studied the different models of teacher leaders that have existed in the district and the associated impact on school professional communities and student mathematics achievement. Our primary approach for studying the impact on school communities has been social network analysis and the approach for studying the impact on student learning has been using value-added assessment models and hierarchical linear modeling.

Research Area 2. Transition to College Mathematics

In comparing placement of MPS to non-MPS graduates in college mathematics courses, we have previously found a significant relationship between the number of high school mathematics units and college placement scores. The MMP piloted, and until 2010 the district required all grade 10-12 students to take, a post-secondary math readiness test. The test was discontinued in 2011, when the school board started requiring all MPS students to take the ACT. We are studying the impact of this and other interventions on high school course enrollments and college math placements.

Research Area 3. Mathematical Knowledge for Teaching (MKT)

During Phase I, we demonstrated significant gains of this specialized knowledge among both pre-service teachers and math teacher leaders. In Phase II, we examined differential gains among pre-service teacher subgroups in their teacher preparation program We also examined the development of mathematical knowledge for teaching among math teacher leaders, and studied the relationship of MKT of classroom teachers and student learning. Organization of Annual Report

Within each research area, we originally identified two or three inquiries, and several specific research questions. During this fourth year, we focused on a subset of our original questions. In Section 1, the relevant inquiries and questions are listed at the start of each subsection where they are discussed; the complete listing of inquires and questions is collected in Appendix A. This section includes a summary of this year’s work, with additional data and results in Appendices B through E. The Implementation Matrix (Exhibit 1) and the Goal Matrix (Exhibit 2) are included in Appendix F. The reference list is contained in Appendix G.

Section 2 provides an overview of our project management. Section 3, our financial report, contains a summary of project expenditures and carryover requests. In Section 4, we present our external evaluator’s report and our response. Finally, in Section 5, we give a brief summary of our anticipated work during the 2013 project year.


5

SUMMARY OF PROJECT ACTIVITIES AND FINDINGS 2012

The Phase II of the MMP engaged in study of three research areas arising from our earlier work. Research Area 1 focused on the district-wide impact of adoption of MMP principles. Research Area 2 is concerned with the transition from high school to post-secondary mathematics. Research Area 3 deals with connections between teachers’ Mathematical Knowledge for Teaching (MKT) and student performance, as well as the effectiveness of our pre-service teacher preparation and in-service professional development in raising teachers’ MKT.

This section discusses our activities and findings during Year 4 of our work, organized by research area and, within each research area, by inquiry. Additional data and results are included in Appendices B through E.

Research Area 1. Mathematics Teacher Leaders and School Impact

In past years, we used data from an online survey, learning team and “math focus” meeting observations, and social network analysis to analyze the effects of different MTL models and school communities on student achievement. We did not repeat these data collection activities in Year 4, choosing to focus our efforts on Inquiry 1a and, in particular, to the link between the adoption of MMP principles and student performance on the Wisconsin Knowledge and Concepts Examination (WKCE)—the state’s annual high-stakes test, administered to all students in grades 3-8 and Grade 10. Appendix B contains additional data and results for this analysis.

Participants

Everyone at the school level who had the potential to positively affect student achievement in mathematics was asked to respond to an online survey, hereafter referred to as the MMP Survey. This year we saw a slight decline in the number that responded to our online survey. The MMP survey was administered from 2009 to 2011. In 2011, there were 1937 MPS school employees who responded to the survey. Table 1 identifies those that participated in the MMP Survey for the three years in which the survey was administered, in terms of their role in the MMP. Table 1. Number of Respondents to the MMP Survey by Role in the MMP

Year 1 (2009) Year 2 (2010) Year 3 (2011) Frequency Percent Frequency Percent Frequency Percent

Math Teacher Leader 147 5.8% 126 6.4% 131 6.8% Learning Team Member & Mathematics Teacher 178 7.0% 281 14.3% 274 14.1%

Learning Team Member & Non-math (e.g., administrative) 234 9.2% 260 13.2% 219 11.3%

Math Teacher Only 1347 53.2% 786 39.9% 842 43.5% Other 624 24.7% 518 26.3% 471 24.3% Total 2530 100% 1971 100% 1937 100%


6

The MMP Survey The primary measures that were considered were the MMP Survey administered in the spring of 2011 and the mathematics subtest of the Wisconsin Knowledge and Concepts Exam (WKCE), administered to students in grades 3 through 8 in the fall of 2011. Responses to the MMP Survey were utilized to create 12 variables that could differentiate schools in terms of the quality of their participation in, and implementation of, MMP related activities. Descriptions of each of the variables created from responses to survey items may be found in Table B-2 of Appendix B, along with the number of items utilized to create each variable and the overall reliability for each subscale obtained from the administration in the spring of 2011 of the MMP Survey. A complete listing of all survey items on the MMP Survey administered in the spring of 2011, compiled by the particular variable each survey item was used to create, can be found in Appendix C. The scores on the 12 variables were then averaged across respondents within a school to come up with overall scores for individual schools. Table B-3 in Appendix B depicts the descriptive statistics for each of these variables from 2009 to 2011. The average scores for each of these subscales were remarkably stable over the three years, with the exception of Engagement 1. Scores on this variable slightly increased each year. This implies that, in general, teachers in MPS were more likely to report working to align their curriculum to MPS standards and using classroom assessments based on standards (CABS). These school level variables were then used as independent variables in subsequent quantitative analyses undertaken to try and predict the student achievement in mathematics in the fall of 2011, as measured by the WKCE. Results from the WKCE assessments were used as a measure of student achievement in mathematics. Scale scores were used for all of our analyses. This is a change from previous reports, in which MMP survey variables were used to predict district benchmark assessment (MAP) growth scores. In these earlier analyses, we found no statistically significant relationships between the MMP Survey and MAP growth. Given this fact, and the fact that the MMP Survey was not administered in the spring of 2012, this year we used MMP Survey variables obtained from the spring of 2011 to predict student achievement on the WKCE in the fall of 2011, in an attempt to evaluate the impact of MMP related activities on increasing student achievement in mathematics.

Results: Value-Added Assessment of District-Wide Impact To what extent can variability in student achievement on the WKCE be attributed to adoption of

MMP principles at the school or classroom level? The first HLM model that was fit to the data utilized student WKCE math scale scores as the dependent variable. No school level predictors were included in the model, resulting in fitting an unconditional model, to determine the total amount of between school level variability that could be predicted by the inclusion of school level variables. These models were fit for two grade level bands: 3-5 and 6-8. The results of fitting these models are depicted in Tables 2 and 3. The variance components in the tables can be utilized to determine the intraclass correlation (ICC), which represents the proportion of variability in student level gain scores that exists between schools, and therefore can be explained by including school level variables in level 2 of the HLM model.


7

For students in grades 3 - 5 the results of fitting an unconditional model are depicted in Table 2. As Table 2 illustrates, the 𝐼𝑅𝐶 = !"#.!"/(!"#.!"!!"#$.!") = 0.15. Therefore, approximately 15% of the variability in student level WKCE scores can be attributed to between school differences for students in grades 3 to 5. Table 2. Results of Fitting Unconditional HLM Model for Grades 3-5 Fixed Effect Coefficient St. Error t Ratio df p-value WKCE (intercept) 432.29 2.22 194.95 100 < 0.001

Random Effect Standard Deviation

Variance Component df c2 p-value

School Mean WKCE Scores 21.82 476.17 100 2422.32 < 0.001 Level 1 Effect 52.05 2709.51

For students in grades 6-8 the results of fitting an unconditional model are depicted in Table 3. As Table 3 illustrates, the 𝐼𝑅𝐶 = !"#.!"/(!"#.!"!!!"#.!") = 0.19. Therefore, approximately 19% of the variability in student level WKCE scores can be attributed to between school differences for students in grades 6 to 8. Table 3. Results of Fitting Unconditional HLM Model for Grades 6-8 Fixed Effect Coefficient St. Error t Ratio df p-value WKCE (intercept) 492.15 2.71 181.70 76 < 0.001




After fitting unconditional models to the data an exploratory analysis was conducted, for schools in each of these two grade level bands, to determine if any of the variables created from the MMP Survey could explain a statistically significant amount of between school variability. The results of the exploratory analysis indicated that Math Focus, Professional Development Quality, Teacher Assessment, Alignment, and Learning Team Quality were all statistically significant predictors of WKCE scores for students in grades 3-5 and grades 6-8. However, there was a very high correlation between Math Focus, which was found to be the strongest predictor, and each of other variables. Therefore, only Math Focus was used in the model to predict student WKCE score. Including any of the other predictors would have resulted in multicollinearity in the model. The results obtained from fitting HLM models, with Math Focus as a predictor, are depicted in Tables B-6 and B-7 in Appendix B. For students in grades 3-5, schools that reported a greater focus on increasing student achievement in mathematics and talking more frequently about the teaching and learning of math with other teachers were more likely to have greater WKCE scores. Specifically, for grades 3-5, the 𝐼𝑅𝐶 = (476.17− 351.99)/476.17 = 0.26. Therefore, approximately 26% of the between school variability for schools that serve students in grades 3-5 can be explained by differences in how focused the school was on increasing students achievement in mathematics.


8

For students in grades 6-8, schools that reported having a greater focus on increasing student achievement in mathematics were more likely to have greater WKCE scores. Specifically, for grades 6-8, the 𝐼𝑅𝐶 = (536.48− 436.87)/536.48 = 0.19. Therefore, approximately 19% of the between school variability for schools that serve students in grades 6-8 can be explained by differences in school focus on increasing students achievement in mathematics.

RESEARCH AREA 2. TRANSITION TO COLLEGE MATHEMATICS

Research Area 2 initially comprised two inquiries: (a) Student readiness for post-secondary mathematics, as evidenced in particular by the MPS Math Readiness Test, and (b) Placement into college mathematics courses. Since MPS has discontinued use of the Math Readiness Test, we have eliminated Inquiry 2a.

Our efforts on Inquiry 2b in Year 4 were focused on analyzing placement data from the Milwaukee Area Technical College (MATC), in an attempt to produce a parallel analysis to that which we have previously done at UWM. Appendix D contains additional data and results for this analysis.

What trends are observed in the proportion of MPS and non-MPS graduates placed in remedial mathematics courses at MATC and UWM?

Placement Trends____________________________________________________________

The data for incoming freshmen over eight years in regards to placing into remedial mathematics courses at UWM and MATC is shown in Table 4. The most notable trend is a narrowing of the remediation gap between MPS and non-MPS students over this time period (from 46.4 to 25.5 percentage points at UWM, and from 45 to 35 percentage points at MATC). Unfortunately, this decline is due more to the increased need for remediation of non-MPS students than to improved placement of MPS graduates. (One point that should be mentioned explicitly is that the apparently significant drop in remedial placement at MATC in Fall 2009 is illusory. This was the first year that MATC was required by the Wisconsin Technical College System to enroll students with particularly low placement scores into an Adult Basic Skills College, rather than admitting them directly into the college. There were 973 of these students in 2009, and they are not included in Table 4.) Table 4. Fall Placement of Incoming Freshmen at UWM and MATC

Placement 2005 2006 2007 2008 2009 2010 2011 2012 UWM Total

MPS 309 269 380 299 375 373 306 266 Non-MPS 3465 3729 4180 3751 3668 3324 3472 3168

Basic Mathematics

MPS 52.0% 46.8% 42.4% 40.1% 43.5% 36.5% 42.2% 41.3% Non-MPS 11.0% 11.0% 11.3% 10.2% 13.1% 12.5% 16.2% 14.2%

Essentials of Algebra

MPS 20.0% 22.3% 30.6% 27.4% 26.7% 33.2% 29.1% 26.9% Non-MPS 14% 19.0% 24.8% 29.1% 26.1% 27.0% 27.0% 28.5%

UWM % Remedial

MPS 71.8% 69.1% 73.0% 67.6% 70.1% 69.7% 71.2% 68.2% Non-MPS 25.4% 30.0% 36.1% 39.3% 39.2% 37.5% 43.2% 42.7%


9

Placement 2005 2006 2007 2008 2009 2010 2011 2012 MATC Total

MPS 798 709 690 721 699 743 986 1037 Non-MPS 528 594 658 637 860 801 729 1430

Basic Mathematics

MPS 72% 74% 75% 73% 66% 70% 69% 62% Non-MPS 26% 29% 25% 27% 24% 28% 50% 30%

Essentials of Algebra

MPS 20% 20% 19% 20% 23% 23% 23% 29% Non-MPS 21% 24% 41% 37% 28% 34% 24% 28%

MATC % Remedial

MPS 92% 94% 94% 93% 89% 93% 92% 91% Non-MPS 47% 53% 66% 64% 52% 62% 74% 58%

MATC Placement Analysis____________________________________________________ During Year 4, we carried out an analysis of MATC placement data, in an attempt to find relationships between students’ scores on the Accuplacer placement test and their high school preparation or ACT scores. Compared with data on student backgrounds for MPS graduates available from the University of Wisconsin-Milwaukee, the data from MATC turned out to be rather incomplete. We did not know the number of units of high school mathematics for any MATC students, and we generally did not have ACT scores, so for the overwhelming majority of MATC students we only had one measurement of their mathematics knowledge prior to beginning their college work. What we did have, in contrast to the UWM data, was the outcome of the first college mathematics course taken at MATC. Overall, and as we shall detail below, outcomes on the Accuplacer Arithmetic Test for all MATC students were poor, and the MPS students were, as a group, less well prepared than the counterparts from other secondary institutions. There were, however, no significant differences between MPS students and students attending other secondary schools in the City of Milwaukee. We had data for students completing high school in 2010, and applying to MATC for Fall 2010 admission. Note that many of these students did not eventually attend MATC. We had similar data for 2009. Between 2009 and 2010, however, MATC changed its course offerings and placement procedures. We will therefore discuss only the 2010 data in this summary; additional details, and the data from 2009, may be found in Appendix D. We have chosen to report data from high schools disaggregated into four categories. MPS Regular refers to MPS schools that are not charter schools or other non-traditional arrangements. Charter schools and similarly administered institutions partnered with MPS are listed as MPS Other. A separate sub-category of Non-MPS schools consists of other schools that are located in the City of Milwaukee. It is important to note that much of the data in this report is what statisticians would call self-reported in the sense that it was provided to MATC voluntarily. As a result, it is very incomplete and no effort should be made to use it to infer what is true about the MATC student population as a whole. For example, we have no way of knowing if the number of Math ACT scores reported is low because students did not report their scores, or because there were no scores to be reported. Similarly, since students are allowed to use a suitable ACT score in place of the Accuplacer, students with higher ACT scores may have opted out of the Accuplacer, depressing the Accuplacer scores. Finally, since MATC has a dual role as a technical school and a two-year liberal arts college, many students may forego mathematics placement testing, irrespective of their mastery of the subject. All of this is in contrast to UWM where both the ACT and the UW System placement test are required of virtually all students entering directly from high school.


10

Accuplacer Data The Accuplacer Mathematics Tests have three components: Arithmetic, Elementary Algebra, and College Level Mathematics. It should be noted that scores on each component range from 20 to 120, and do not directly reflect the number of correct responses as the test is given in a computer adaptive environment. For each test component, we have grouped scores in ranges. The meaning of each range for each test segment is described in Appendix D. Here, we will only note that students scoring below 93 on the Arithmetic segment do not indicate readiness for college work involving even the most rudimentary mathematics skills, while a score of 109 or higher on the Elementary Algebra segment corresponds roughly to the successful completion of MATH 095, and placement into MATH 105 or other credit-bearing courses, at UW-Milwaukee. MATC has multiple courses at the level of MATH 095 at UWM, including MATGEN 110. Table 5 below shows the raw numbers on the Accuplacer Arithmetic segment, for students applying for admission to MATC in 2010. The first observation from Table 5 is that we have Accuplacer scores for only 2186 students out of a total of 3715 (59%). Of those 2816, only 153 (7%, or 4% of all students) scored 93 or higher. Figures 1 and 2 show percentages for all students, and for students for whom we have Accuplacer results. Table 5. 2010 MATC Accuplacer Arithmetic Scores

All Students MPS Regular MPS Other Not MPS CITY, Not MPS No ACT, No ACC 1119 479 76 564 64 ACT, No ACC ARI 410 67 6 337 27 20- 37 1044 561 125 358 50 38- 64 583 204 46 333 24 65-92 406 99 21 286 8 93-109 117 31 5 81 4 110-120 36 4 1 31 2 Total 3715 1445 280 1990 179

Figure 1. 2010 MATC Accuplacer Arithmetic Percentages, All Students

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

110-‐120

93-‐109

65-‐92

38-‐ 64

20-‐ 37

ACT, No ACC ARI

No ACT, No ACC


11

Figure 2. 2010 MATC Accuplacer Arithmetic Percentages, Students with Accuplacer Scores

Table 6 below shows the raw numbers on the Accuplacer Elementary Algebra segment, for students applying for admission to MATC in 2010. The first observation from Table 5 is that we have Accuplacer scores for only 2186 students out of a total of 3715 (59%). Of those 2816, only 153 (7%, or 4% of all students) scored 93 or higher. Figures 3 and 4 show percentages for all students, and for students for whom we have Accuplacer results. Table 6. 2010 MATC Elementary Algebra Accuplacer Scores

All Students MPS Regular MPS Other Not MPS CITY, Not MPS No ACT, No ACC 2708 1226 238 1244 132 ACT, No ACC ALG 452 81 13 358 32 20- 27 34 16 2 16 2 28- 43 135 47 10 78 0 44-81 292 61 16 215 10 82-108 87 14 1 72 3 109-120 7 0 0 7 0 Total 3715 1445 280 1990 179

We have relatively little data on this segment. This is not surprising given the adaptive nature of the Accuplacer: if students score poorly on the Arithmetic portion they are not examined on Elementary Algebra. We have 555 results for Elementary Algebra, and 2186 results for Arithmetic, and all but 559 of those 2186 students scored in our bottom two categories on Arithmetic. Recall that a score of 109 or higher on the Elementary Algebra segment of the test should correspond to placement into a credit-bearing course at UWM. Only 7 of the 555 students for

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

All Students

MPS Regular

MPS Other

Not MPS CITY, Not MPS

110-‐120

93-‐109

65-‐92

38-‐ 64

20-‐ 37


12

whom we have an Elementary Algebra score (1.3%) scored at this level, and none of those were from MPS. Since we know from Table 4 that 8% of MPS students were placed into credit-bearing courses at MATC in 2010, we hypothesize that relatively stronger students are using their ACT scores for placement at MATC, and choosing not to take the Accuplacer. We have no way at present of verifying this hypothesis, however. Figure 3. 2010 MATC Accuplacer Elementary Algebra Percentages, All Students

Figure 4. 2010 MATC Accuplacer Elementary Algebra Percentages, Students with Accuplacer Scores

Only 260 students of our cohort of 3715 students (7%) reported any score in the College Level Mathematics category. Given the adaptive nature of the test, and the results reported above for Arithmetic and Elementary Algebra, this was to be anticipated, but we did not analyze this category further, as the amount of data was insufficient to draw meaningful conclusions.

0%

20%

40%

60%

80%

100%

All Students MPS Regular MPS Other Not MPS CITY, Not MPS

No ACT, No ACC ACT, No ACC ALG 20-‐ 27 28-‐ 43

44-‐81 82-‐108 109-‐120

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%


20-‐ 27 28-‐ 43 44-‐81 82-‐108 109-‐120


13

ACT Data From our 2010 cohort of 3715 students, 494 (13%) reported ACT mathematics scores. Since MATC does not require the ACT for admission, we do not know how many of the students in this study have taken the ACT. There is an incentive to report scores, as they may be used in place of the Accuplacer for placement decisions. In future years there should be a significant rise in the numbers of ACT Math scores reported by MPS students as well as now there is a district-wide policy that taking this test is a district expectation. An interpretation of ACT scores may be found in (Key & O’Malley, 2004), or at the ACT website, http://www.act.org/products/k-12-act-test. Table 7 below shows the raw numbers for students applying for admission to MATC in 2010; Figures 5 and 6 show the percentages for all students, and for those students reporting scores. Table 7. 2010 MATC Mathematics ACT Scores

All Students MPS Regular MPS Other Not MPS CITY, Not MPS No ACT 3221 1352 267 1602 146 1-12 3 1 0 2 1 13-15 33 15 3 15 3 16-19 249 59 10 180 13 20-23 133 13 0 120 14 24-27 64 5 0 59 2 28-32 11 0 0 11 0 33-36 1 0 0 1 0 Total 3715 1445 280 1990 179

Figure 5. MATC Mathematics ACT Scores, All Students

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


No ACT 0-‐12 13-‐15 16-‐19 20-‐23 24-‐27 28-‐32 33-‐36


14

Figure 6. MATC Mathematics ACT Scores, Students with Mathematics ACT Scores

We see that more students reporting scores have a score in the 16-19 range. According to the interpretation of ACT scores in (Key & O’Malley, 2004), students scoring in this range should be considered ready to take MATH 095 at UWM; thus, a student with a score of 19 or less would not be considered ready for a credit-bearing college mathematics course. Aggregating the data for MPS-Regular and MPS-Other in Table 7, we find that 88 out of 106 (83%) MPS students reporting ACT scores had scores of 19 or less, whereas 197 out of 388 (51%) of non-MPS students reporting ACT scores had scores of 19 or less. These results are broadly consistent with the data in Table 4 (93% and 62% remedial placement respectively), but suggest that students’ mathematics skills might be improving between the time they take the ACT and the time they apply for admission to MATC. Course Results We have the results for mathematics coursework for 837 students who enrolled in mathematics courses at MATC in the Fall of 2010: 615 students who enrolled in non-credit courses, and 222 who enrolled in credit-bearing courses. Success in this coursework is, at a minimum, an indication that these students are placed correctly and take their coursework seriously. MATC has several non-credit mathematics courses, and the distinction between them is not always clear. We will aggregate certain courses, which appear to contain elementary and middle school mathematics, under the labels MathB and MathPH respectively, and then consider the courses MatGen 109 and MatGen 110 separately. For comparison, MATGEN 109 and MATGEN 110 are each roughly equivalent to MATH 095 at UWM. No one credit-bearing course had sufficient enrollment to allow us to make any credible judgment as to student success, so we have grouped those courses together.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


0-‐12 13-‐15 16-‐19 20-‐23 24-‐27 28-‐32 33-‐36


15

Note that MATC does not use the grade of ‘F’, using “Unsatisfactory” in its place. MATC also uses the grade of ‘E’, indicating effort. This is neither a passing nor a failing grade. Finally, no explanation has been provided for why there are students listed as having enrolled in a course but for whom no grade is listed, given that a “W” indicates that a student has withdrawn from the course. Non-Credit Courses Tables 8 and 9 below show the results for students enrolling in MathB (elementary school mathematics) and Math PH (middle school mathematics). There is not much data here to analyze, but it is clear from Table 8 that for this group of students, the MPS students who attended MPS high schools were not as well prepared, even for elementary school mathematics, as their counterparts who attended non-MPS high schools. To be specific, 48 out of 142 MPS students (34%) earned a passing grade in MathB, whereas 70 out of 131 non-MPS students (53%) did so. Table 8. 2010 MathB Results All Students MPS Regular MPS Other Not MPS A/A- 2 0 0 2 Pass 116 40 8 68 E 18 14 0 4 Unsatisfactory 16 10 1 5 Withdraw 55 27 4 24 No Grade 66 36 2 28 Total 273 127 15 131

Table 9. 2010 MathPH Results All Students MPS Regular MPS Other Not MPS B+ 1 0 0 1 C 1 1 0 0 D 2 2 0 0 Pass 11 8 1 2 E 1 0 0 1 Unsatisfactory 1 0 0 1 Withdraw 5 3 0 2 No Grade 5 2 0 3 Total 27 16 1 10

Tables 10 and 11 show the results for students enrolling in MATGEN 109 and MATGEN 110. Out of the 77 MPS students enrolled in MATGEN 109, 33 (43%) earned a grade of C- or better, whereas 92 out of 161 (57%) Non-MPS students did so. The number of students enrolled in MATGEN 110 is too small to draw any real conclusions, but it does appear from Table 11 that MPS students who enrolled in the course again received generally lower grades than non-MPS students. MATGEN 109 and 110 are roughly equivalent to Math 095 at UWM, where the success rates are a bit higher, but not radically so.


16

Table 10. 2010 MATGEN 109 Results All Students MPS Regular MPS Other Not MPS A/A- 46 11 1 34 B/B+ 46 12 2 32 C/C+/C- 33 6 1 26 D/D- 9 3 1 5 Unsatisfactory 14 7 0 7 Withdraw 34 13 4 17 No Grade 56 14 2 40 Total 238 66 11 161

Table 11. 2010 MATGEN 110 Results All Students MPS Regular MPS Other Not MPS A/A- 12 0 0 12 B/B+ 13 2 0 11 C/C+/C- 14 1 0 12 D/D- 9 4 0 5 Unsatisfactory 4 1 0 3 Withdraw 9 4 2 3 No Grade 17 1 0 16 Total 77 13 2 62

Credit-Bearing Courses Table 12 below shows the results for students enrolling credit-bearing courses. Once again, the number of MPS students is too small to draw any meaningful conclusions. Table 12. 2010 Credit-bearing Course Results All Students MPS Regular MPS Other Not MPS A/A- 21 4 0 17 B/B+ 38 4 0 34 C/C+/C- 37 0 0 37 D/D- 22 2 0 20 Unsatisfactory 20 4 0 16 Withdraw 30 8 2 20 No Grade 54 5 3 46 Total 222 27 5 190

Delayed-Entry Students MATC enrolls many non-traditional students. This provides us with an additional pool of students whose academic progress we may examine. In this subsection, we will examine the progress of students who delayed entry by one year; that is, students who completed high school in 2009 and applied to MATC for admission in 2010. We have identified 1240 such students. For this analysis, we will only separate out the MPS Regular students, combining charter students with the Non-MPS group. Table 13 below shows the Accuplacer scores for our cohort of delayed-entry students.


17

Table 13. 2010 Delayed-entry Accuplacer Results All Students MPS Regular Not MPS No ACT, No ACC 278 93 185 ACT, No ACC ARI 104 7 97 20- 37 371 206 165 38- 64 257 114 143 65-92 158 40 118 93-109 58 12 46 110-120 14 1 13 Total 1240 473 767

As shown in Table 14 below, we have 134 Math ACT scores for these students. Most of these scores are from students not attending MPS. By comparison with the students who did not delay entry by one year, these students are very poorly prepared, with the MPS students worse off. Only 52 of the 767 non-MPS students took a credit-bearing course, and only 16 of the 473 MPS students did so. As with the direct entry students, there is a glaring disparity between the two groups when it comes to presenting ACT scores. This disparity should disappear in future years, as pointed out earlier, since MPS has instituted new policies regarding the ACT. Table 14. 2010 Delayed-entry Mathematics ACT Results All Students MPS Regular Not MPS No ACT 1106 459 647 13-15 6 1 5 16-19 63 9 54 20-23 39 4 35 24-27 21 0 21 28-32 5 0 5 Total 1240 473 767

Table 15 below gives the grades earned by the delayed-entry students who took any mathematics course at all. For these delayed-entry students, 71 out of 262 (27%) MPS students earned a grade of C- or better, whereas 58 out 168 (35%) of non-MPS students did so. These percentages are considerable lower than those of students who graduated high school in 2010. Neither group of delayed-entry students completed their mathematics courses successfully, with the MPS students faring worse than their non-MPS counterparts. Table 15. Delayed-entry Mathematics Course Results All Students MPS Regular Not MPS A/A- 18 1 17 B/B+/B- 24 4 20 C/C+/C- 29 8 21 D/D-/D+ 11 6 5 Pass 28 8 20 E 7 5 2 Unsatisfactory 20 15 5 Withdraw 55 22 33 No Grade 70 25 45 Total 262 94 168


18

Research Area 3. Mathematical Knowledge for Teaching (MKT)

To study the impact on teachers’ mathematical knowledge for teaching (MKT), we are using measures from the Learning Mathematics for Teaching project at The University of Michigan. These items estimate ability in using mathematical knowledge in the context of teaching. The measures utilize item-response-theory (IRT) to evaluate items, construct scales, and report results. This third research area is comprised of three inquiries: (1) the impact of the UWM mathematical preparation program on the development of MKT; (2) the impact of mathematics content-based professional development on the MKT of mathematics teacher leaders; and (3) the relationship of teacher MKT to student achievement. The full report on our examination of MKT is found in Appendix E. Here we present a summary of our work and findings. Inquiry 3a. Impact of Mathematical Preparation on Pre-service Teachers’ MKT

Do prospective teachers with a mathematics minor have stronger MKT than those without?

To what extent can variability in MKT be attributed to mathematical preparation (i.e., math foundation courses, math minor courses, methods courses, practicum experiences)?

What is the relationship of MKT to other measures (e.g., Praxis scores) and to teaching practice?

At the core of effective STEM teaching in the area of mathematics is deep knowledge of the content that is relevant to teaching and the habits of mathematical practice. Effective teachers are well aware of mathematical progressions and of the coherence of mathematics. Teachers with deep content knowledge can successfully help students understand complex ideas and make mathematical connections and applications, and help students develop mathematical proficiency. Content knowledge development begins in preservice teacher preparation with rigorous coursework aimed at ensuring that new teachers have the knowledge and skills needed to effectively begin their careers teaching mathematics.

The Milwaukee Mathematics Partnership (MMP), as one of its core initiatives, redesigned the mathematical preparation of teachers at the University of Wisconsin-Milwaukee. This work was based on recommendations highlighted in The Mathematical Education of Teachers (hereafter, MET) (CBMS, 2001) and Adding It Up (NRC, 2001). A vital aspect of the approach was the use of design teams comprised of a mathematician who was responsible for ensuring the course contained rigorous and correct mathematics; a mathematics educator who ensured the course aligned with current educational thinking and mathematics curricula; and a Teacher-in-Residence, who ensured course material related to classroom practice. To that end, the MMP revised the foundational mathematics content courses required of all prospective elementary and early childhood preservice teachers. Then developed four new mathematics courses for preservice elementary teachers: (1) Mathematical Problem Solving, (2) Geometry, (3) Discrete Probability and Statistics, and (4) Algebraic Structures.


19

The development of the four new mathematics courses was of particular importance given that the university had just taken an extraordinary step to require all elementary education majors to choose a minor in mathematics or science, along with a second minor in social studies or English/language arts. In the past, when only one minor was required, most majors chose social studies and less than 8% chose mathematics. While the minor requirement had changed, the curriculum had not and education majors at first just elected from the existing array of mathematics courses. Thus, a goal of the MMP was to develop new courses specifically for the minor that built and deepened the content knowledge needed to teach mathematics in alignment with the MET and NRC recommendations and the findings of Ma (1999) and Ball and colleagues (Ball, 2003; Ball & Bass, 2003; Ball, Thames, & Phelps, 2008; Hill, Rowan, & Ball 2005). The 18-credit mathematics minor was in addition to two foundational mathematics courses for a total of 24 credits in mathematics content, in addition to six credits of mathematics methods. The minor also included existing coursework in calculus concepts. The four new courses are now permanent course offerings as part of the elementary education mathematics minor. The central question of interest is whether or not preservice teachers enrolled in these courses have, in fact, developed deep mathematics content knowledge necessary for effectively beginning their careers as STEM teachers.

To study the impact on preservice teachers’ mathematical knowledge for teaching (MKT), we used measures from the Learning Mathematics for Teaching (LMT) project at The University of Michigan (Hill, Ball, & Schilling, 2008; Hill, Sleep, Lewis, & Ball, 2007). These items estimate ability in using mathematical knowledge in the context of teaching. The measures utilize item-response-theory (IRT) to evaluate items, construct scales, and report results. We constructed our own scales from the bank of items and analyzed the data using a two-parameter model.

In our longitudinal study, we tracked math course enrollment and measured the MKT of preservice elementary teachers at four time points as they progressed through their teacher education program. The first MKT provided a baseline at the start of their mathematical foundation courses. The second MKT was measured after completion of the foundation courses and, most if not all, of the courses for the mathematics minor, but prior to taking the mathematics methods courses. The third MKT was taken at the beginning of the mathematical methods courses. The fourth MKT was completed at the conclusion of the mathematical methods courses. We were then able to compare the MKT of the elementary education major with a mathematics minor to those without a math minor. We hypothesize that teachers electing a mathematics minor and who had taken the new mathematics content courses would demonstrate stronger MKT than those not electing the minor. This comparison group design allowed us to test this hypothesis with the goal of determining the effectiveness of the new content courses for deepening mathematical knowledge for teaching. In addition, we also tracked and measured the MKT of early childhood majors as an additional comparison group. Sample_________________________________________________________________ The sample for this analysis focused on students who (1) had completed their pre-service education and (2) had taken the MKT assessment at all four points in time for either Number and Operations or Geometry. This produced a sample of 315 preservice teachers. Table 16 indicates the breakdown of this sample.


20

Table 16. Sample Breakdown by Program Area

Program Frequency Percent

MCEA Math 75 23.8

MCEA Other 101 32.1

Early Childhood 134 42.5

Other Program 5 1.6

Total 315 100.0 Program Progression_______________________________________________________ The first question of interest focuses on the courses taken by different groups of teachers throughout their preservice mathematics preparation. To assess this, we examined enrollment data for our sample of 315 preservice teachers, as well as ACT mathematics scores and Praxis II scores (this exam is taken at the end of the program prior to licensure). In examining enrollment data, we were interested in the proportion of our sample who had enrolled in non-credit mathematics courses (Math 90/95) as well as the proportion who had enrolled in mathematics focus courses (Math 275, 276, 277, and 278). Table 17 displays these results. Table 17. Mathematics Course Enrollment by Program

Program

MCEA Math MCEA

Early Childhood Other Mean

N 75 101 134 5 315 ACT Math Score 23.46 20.37 19.76 19.00 20.83 Math 090 0% 13% 14% 20% 10% Math 095 7% 31% 43% 60% 30% Math 175 92% 96% 96% 100% 95% Math 176 100% 100% 100% 100% 100% Math 275 91% 2% 0% 0% 22% Math 276 40% 0% 0% 0% 10% Math 277 99% 1% 0% 0% 24% Math 278 31% 0% 0% 0% 7% Praxis II Score 164.05 161.76 160.04 162.65

The MCEA Math students stand apart from other preservice teachers in several ways. First, their average ACT Mathematics score is over 3 points higher than the scores for other MCEA students or Early Childhood students. Second, only a small proportion (7 percent) of MCEA Math students was required to enroll in non-credit mathematics courses compared with significantly higher proportions of other MCEA students (31 percent) and Early Childhood students (43 percent). This indicates that the overall mathematics competency of the MCEA Math students was higher upon entering college.


21

Finally, and as expected, high proportions of MCEA Math students enrolled in the mathematics focus courses compared with negligible or no enrollment by students in other programs. This is as intended given the focus courses were specifically designed with this group of students in mind. Descriptive and Repeated Measures Analysis_____________________________________ Descriptive MKT results for Number and Operations and for Geometry (see Appendix E, Tables E-3 and E-4) show that for each content area, MCEA Math minors started their program with higher MKT than other MCEA and Early Childhood preservice teachers. MCEA Math minors also made strong progress throughout their program, increasing their MKT in each content domain by the end of the required mathematical methods courses. It is also important to note that MCEA and Early Childhood preservice teachers also increased their MKT throughout their program showing strong overall improvement for number and operations and geometry. It is also interesting to note that smaller gains were made by the MCEA Other and Early Childhood preservice teachers following the foundation courses for number and operations and that the Early Childhood preservice teachers had a decrease in MKT for geometry. Given these descriptive results, we were interested in determining if MCEA math minors and other MCEA students made gains throughout their programs and if there were any differences in the gains demonstrated by these groups of students. The repeated measures analysis reported below used MKT assessment score as the within-subjects factor and MCEA program (math minor or non-math minor) as the between subjects factor. Figure 7 displays the descriptive results from this analysis for the number and operations MKT assessment. Note that both groups made gains but it appears that MCEA math minors made greater gains. What is most apparent are the gains made by MCEA math minors between the end of the foundation courses and the start of the methods courses when compared with the slight decline in MKT demonstrated by other MCEA students. Figure 7: Number and Operations Overall Program Impact

175 Pre-‐Test 175 Post-‐Test 331 Pre-‐Test 332

Post-‐TestMCEA Math -‐.28 .05 .14 .46Other MCEA -‐.65 -‐.21 -‐.31 -‐.02All MCEA -‐.49 -‐.10 -‐.12 .19

-‐.80

-‐.60

-‐.40

-‐.20

.00

.20

.40

.60

MKT Score


22

Repeated measures analysis for number and operations indicated a within subjects effect (F=39.35, df=3, p=.000) but did not indicate an interaction effect between MCEA program and MKT scores. Analysis of between subjects effects showed that there was a difference in the level of MKT scores between MCEA math minors and other MCEA students (F=13.73, df=1, p=.000). From this analysis, we conclude that (1) both groups of MCEA students made statistically significant gains in MKT scores throughout their program, (2) that MCEA math minors have higher levels of MKT, and (3) the gains demonstrated by each group of students were not statistically different from each other. Figure 8 displays the results for the geometry assessment. As with the number and operations results, MCEA math minors demonstrated higher MKT from the beginning all the way through their program. MCEA math minors also appeared to show greater gains throughout their program when compared with the other MCEA students. Also notable is the decline in MKT shown by these other MCEA students from the end of the foundation course to the beginning of the methods courses. Figure 8: Geometry Overall Program Impact

Repeated measures analysis for geometry indicated a within subjects effect (F=29.67, df=3, p=.000) as well as an interaction between MCEA program and MKT scores (F=3.47, df=3, p=.016). Analysis of between subjects effects showed that there was a difference in the level of MKT scores between MCEA math minors and other MCEA students (F=16.26, df=1, p=.000). This is clearly shown in Figure 8. From this analysis, we conclude that (1) both groups of MCEA students made gains in MKT scores throughout their program, (2) that MCEA math minors have higher levels of MKT, and (3) the gains demonstrated by each group of students were statistically different from each other; i.e., program selection did make a difference in geometry MKT scores.

176 Pre-‐Test 176 Post-‐Test 331 Pre-‐Test 332 Post-‐

TestMCEA Math -‐.11 .34 .37 .49Other MCEA -‐.34 .07 -‐.10 -‐.02All MCEA -‐.24 .18 .09 .19

-‐.40-‐.30-‐.20-‐.10.00.10.20.30.40.50.60

MKT Score


23

Figure 9 displays results for the Early Childhood students for Number and Operations and Geometry. As shown, Early Childhood students showed gains in both their number and geometry content knowledge. Those gains were steady for the Number and Operations scale. For the Geometry scale, their knowledge increased by the end of the foundation courses and then shows some decline by the end of the methods course, suggesting a lesser focus on Geometry in the methods course. However, their final geometry score at the end of the methods course was still higher than their initial pretest score. Figure 9. Early Childhood Program Impact

Repeated measures analysis for Early Childhood students for Number and Operations indicated a within subjects effect (F=9.89, df=2, p=.000). Similarly, a within subjects effect was found for Geometry scores (F=13.08, df=2, p-.000). From this analysis, we conclude that Early Childhood students made significant content knowledge gains during their program in both areas. Impact of Mathematics Focus Courses__________________________________________ The results of the repeated measures analysis indicated that MCEA math minors made gains in MKT or held steady from the end of their foundation courses until the beginning of the mathematics method courses. This was not the case for other MCEA students who did not elect a math minor. These findings suggest that the additional mathematics preparation through the mathematics focus courses made a difference in preservice teacher MKT. There are four mathematics focus courses for preservice elementary teachers: (1) Math 275 Mathematical Problem Solving, (2) Math 277 Geometry, (3) Math 278 Discrete Probability and Statistics, and (4) Math 276 Algebraic Structures. Our approach was to use regression analysis to determine if enrollment in one or more of the math focus courses helped predict performance on the final MKT assessment for either the Number and Operations scale or the Geometry scale. Given the differences in MKT between MCEA Math minors, other MCEA students, and Early

175 Pre-‐Test 175 Post-‐Test 330 Post-‐TestNumber and Operations -‐.71 -‐.50 -‐.38

Geometry -‐.43 -‐.10 -‐.22

-‐1.00-‐.90-‐.80-‐.70-‐.60-‐.50-‐.40-‐.30-‐.20-‐.10.00

MKT Score


24

Childhood students, we would hypothesize that enrollment in the math focus courses did make a difference in MKT. The first result of this analysis (see Appendix E, Table E-5) was that the Math 175 post-test score (t=8.61, p=.000) and enrollment in Math 277, Geometry for Elementary Teachers (t=3.19, p=.002) were significant predictors of the final Number and Operations MKT Assessment Score. This is an important finding that suggests enrollment in at least one of the math focus courses, Math 277, is an important predictor of preservice teacher MKT. The second result of the analysis (see Appendix E, Table E-6) was that the Math 176 post-test score (t=9.23, p=.000) was a significant predictor of Final Geometry MKT Assessment score. Enrollment in Math 276, Algebraic Structures for Elementary Teachers was nearly significant (t=1.71, p=.088). This suggests that the most important predictor of final MKT Geometry performance is performance in the Math 176 course, and that enrollment in the focus area courses does not contribute significantly to final performance. Conclusions___________________________________________________________________ Overall, preservice teachers were demonstrating gains in mathematical knowledge for teaching as measured by the MKT assessments administered throughout their program of study. Results indicated that MECA preservice teachers with a math minor have stronger MKT scores than those who did not elect a math minor. For students who have completed their program, the elementary teachers with a math minor began and ended their program with higher MKT scores than the non-math elementary teachers and the early childhood teachers on measures of Number and Operations and Geometry. These results may be due to self-selection bias where individuals who were more successful in mathematics, generally, scored better on the MKT assessments and self-selected a math minor. This assertion is supported by the higher ACT math scores of students electing a mathematics minor when compared with other students.

An alternative explanation is that the additional math content courses taken by those electing a mathematics minor may explain the greater gains exhibited by these students. This appears to be true for the Number and Operations assessment where enrollment in Math 277 clearly predicted MKT assessment scores. This was not necessarily true, however, for the Geometry assessment where enrollment in math focus courses did not predict final Geometry assessment scores. A final important question is how does the MKT of preservice teachers at the end of their program, compare with that of inservice teachers and even that of Math Teacher Leaders (MTLs) in the Milwaukee Public Schools. To evaluate this question, we used data collected over time from classroom teachers and MTLs and compared that to our sample of 315 program completers, broken out by academic program, used in the analysis throughout this report. Table E-7 displays the descriptive statistics for this analysis. A one-way analysis of variance was conducted to determine if there were statistically significant differences in the MKT assessment scores among MCEA Math preservice teachers, MCEA Other preservice teachers (non-math), Early Childhood preservice teachers, district classroom teachers, and district Math Teacher Leaders. For Number and Operations, the one-way ANOVA indicated significant differences across the five groups (F (4,1128)=52.77, p=.000). Post-hoc


25

analysis revealed that MCEA Math Minors scored significantly higher than the other groups, with the exception of Math Teacher Leaders, with whom they scored similarly.

For Geometry, the one-way ANOVA indicated significant differences across the five groups (F(4,1162)=21.19, p=.000). Post-hoc analysis revealed that MCEA Math Minors scored significantly higher than the other groups, with the exception of Math Teacher Leaders, with whom they scored similarly. Table E-7: Descriptive Statistics for MKT Comparisons

MKT Assessment

Program/Group

MCEA Math MCEA Early

Childhood Classroom Teachers

Math Teacher Leaders

Number and Operations

Mean .46 -.02 -.38 -.30 .50 SD .65 .67 .60 .76 .93 N 51 67 99 713 203

Geometry Mean .49 -.02 -.22 -.14 .32 SD .70 .70 .52 .80 .94 N 60 88 103 713 203

This final analysis is important—it demonstrates that the mathematics preparation received by mathematics minors during their preservice teacher education compares favorably with the professional development received by Math Teacher Leaders throughout the MMP program. Similarly, preservice math minors can be expected to have stronger mathematics preparation than other preservice teachers as well as the typical classroom teacher in the Milwaukee Public Schools. All of this suggests that the MMP has been extremely successful in improving the level of preservice teacher preparation at the University of Wisconsin-Milwaukee. Inquiry 3b. Impact of Content Development on Math Teacher Leaders’ MKT

What is the impact of partnership-driven professional development on the mathematical knowledge for teaching of math teacher leaders?

What is the relationship of teacher leader MKT to other measures (e.g., professional development hours, school math focus, student achievement) and to teacher leader practice?

Important changes in the MTL program were made for the 2011-2012 school year. Most significantly, the number of MTLs supported by the school district fell from approximately 115 to fewer than 50. Some of these were tenured MTLs and others were new to the role. Each of these MTLs was assigned to support 3-4 different schools, rather than a single school. This shift in program design created a situation where some MTLs were new to the role given that tenured MTLs did not want to continue in the role given the new program design. At the same time, partnership-driven professional development of Math Teacher Leaders (MTLs) continued during the 2011-2012 school year. This development focused on improving MTL content knowledge in the domain of number and operations. This included revisiting topics studied earlier during the MMP given the numbers of individuals new to the MTL position.


26

Throughout the MMP program, MTLs have taken the MKT Assessment at various points in time. The first assessment for the Phase II work was administered in Fall 2008. Subsequent assessments were administered in Spring 2009, Spring 2010, Spring 2011, and Spring 2012. Each year there were some new MTLs. These individuals also took a pre-assessment in the Fall of the year in which they first became an MTL. The objective of this inquiry was to evaluate MTL content knowledge gains as measured by the MKT assessment, as well as to detect any content knowledge differences between different groups of MTLs. Our approach to this analysis was to examine the differences between pre-assessments and post-assessments for a given year. Descriptive and Paired Sample results are presented in Tables E-8 through E-12 of Appendix E. Sample Our sample contained 300 unique MTLs with an average tenure of 3.43 years and a median tenure of 3 years (SD=2.20). Figure 10 shows the breakdown of the sample by years as an MTL. Nearly half of the sample of MTLs (44 percent) had been an MTL for two or fewer years. This suggests regular turnover of MTLs throughout the four-year MMP Phase II period. Figure 10. MTL Sample by Years as an MTL

Results Partnership-driven professional development has been a significant component of the MMP since its inception. Each year, MTLs focused on a particular content knowledge strand and were tested to determine if their content knowledge improved.


27

Overall, results suggest that MTLs made strong content knowledge gains in previous years, but that improvement has slowed, or even reversed, in recent years. For example, in the 2008-2009 school year, MTLs demonstrated improvement on the Geometry and Algebra scales of the MKT assessment. However, by Spring 2011, the overall level of Geometry knowledge demonstrated by MTLs was quite a bit lower than previously displayed. The likely explanation for this was turnover in the MTL population. We know that in recent years, MTLs were less tenured than in previous years. In Spring 2009, the average tenure of an MTL in our sample was 4.5 years. By Spring 2012, the average tenure had fallen to 4.1 years. To test this conjecture, we examined the relationship between MTL tenure and MKT assessment results. Interestingly, we did not find a strong relationship between these two indicators suggesting that MTL tenure (i.e., accumulated professional development over many years), does not necessarily lead to stronger MKT. What then, might be the cause of changes in MKT assessment scores. First, the change in MTL model may have an impact. Most recently in 2011-2012, MTLs shifted from supporting a single school to supporting multiple schools. This may have detracted from their own professional development and provided limited opportunities for MTLs to utilize the content knowledge they were studying with teachers across their schools. Given this new model, many MTLs noted that they devoted considerable time to building relationships and touching on a myriad of topics in response to needs of individual teachers and schools. Thus, their work was more diffused and less focused on the content strand of study as was the case in previous years. Second, there has been no measure of the quality of professional development delivered but MKT results suggest that it has degraded slightly as the Milwaukee Public Schools District has taken on greater responsibility for the professional development and University of Wisconsin-Milwaukee personnel have taken a lesser role. Despite these changes and inconsistent results, though, it is clear that the professional development has been impactful for many MTLs. The sample size alone—300 MTLs—suggest that impact has been broad and likely beneficial to many across the Milwaukee Public Schools District. Inquiry 3c. Linking Teacher MKT to Student Achievement

What is the relationship between teacher MKT and student achievement?

Is there a stronger relationship between teacher MKT and (a) student knowledge gains throughout the school year as measured by district benchmark exams or (b) student achievement

as measured by the state standard test administered the subsequent fall? Teacher mathematical knowledge for teaching (MKT) has been found to be related to student achievement (Hill, Rowan & Ball, 2005; Hill & Ball, 2004). An aspect of our work in Phase II was measuring teacher MKT to contribute to the body of knowledge about the relationship between teacher MKT and student achievement. Specific questions for this analysis were:


28

1. What is the relationship between teacher MKT and student achievement growth? 2. In addition to teacher MKT, what other teacher demographics might help explain

variation in student achievement growth? 3. In addition to teacher demographics, what student demographics might help explain

variation in student achievement growth? Data Sources Data have been compiled over a six-year period. Teacher MKT and demographic information were obtained from classroom teachers during the spring of each year from 2006 through 2011. The MKT assessment was scored and IRT ability estimates were computed for each MKT sub-scale—Number and Operations, Algebra, Geometry, and Probability and Statistics—and overall. The overall MKT assessment score and teacher demographics were used as teacher level predictors. Student achievement data were obtained from the Milwaukee Public Schools (MPS) through our partnership with the MPS Department of Research. We used data from the WKCE exam from 2005 through 2011. We calculated gain scores for individual students who took the WKCE during this period. To obtain a gain score, a student had to take the MKT in two consecutive years. Student demographics were also compiled from the MPS data.

The critical challenge for this analysis was how to link teachers to students. For this analysis, we linked teachers to the students they had during the school year in which they took the MKT. For example, a teacher who took the MKT in Spring 2006 was linked to the students he or she taught during the 2005-2006 school year.

The corresponding student gain score used in the analysis was calculated by subtracting the Fall 2005 WKCE scale score from the Fall 2006 scale score. Similarly, a teacher taking the MKT in Spring 2011 was linked to students he or she taught during the 2010-2011 school year and the corresponding student gain score was the difference between the student’s Fall 2010 and Fall 2011 achievement scores. Using this approach allowed us to substantially boost the sample size for our analysis by incorporating multiple years of data into the analysis. HLM Analysis Results

Hierarchical Linear Modeling (HLM) was used to estimate the proportion of variation in student achievement gain scores that could be attributed to (1) teachers overall, (2) teacher MKT scores, and (3) other teacher demographic variables. HLM was used because student achievement results were nested within teacher MKT scores. HLM is the appropriate analysis to partition variance in a dependent variable (student achievement) according to multiple levels of factors (student level factors and teacher level factors). Several models were fit to the data. First, an unconditional model was fit to determine how much variability in the student achievement growth could be attributed to between teacher variability. If the unconditional model was non-significant, there was no value in continuing the analysis since that would mean that between teacher differences did not account for variability in student


29

outcomes. The results from the unconditional model indicated that 15.5 percent of the variability in student scale score growth could be attributed to teacher level variables. Thus, it was worthwhile to fit a conditional model with teacher MKT and other demographics as predictor variables. An exploratory analysis using Level 2 (teacher) variables as potential predictors of variation in achievement growth indicated that potential Level 2 predictors were primary teaching responsibility (i.e., teaching level) and MKT assessment scores. We fit a first conditional model, using primary teaching level and MKT assessment results at the teacher level to determine if they helped explain a statistically significant proportion of the variability in student achievement growth. The results of fitting this conditional model (see Appendix E, Table E-18) indicated that primary teaching responsibility does help explain additional variation in WKCE scale score growth year over year. The amount of variability explained was significant: 31.2 percent. \ It is important to note that the coefficient was negative indicating that primary teaching responsibility was a negative predictor. This indicates that student growth must slow at higher grade levels (for example from Grade 6 to Grade 7, or Grade 7 to Grade 8). Thus the relationship between primary teaching responsibility and student achievement growth is negative. It was also found that teachers teaching at higher grade levels demonstrated stronger MKT assessment scores. Thus, teacher MKT was not a significant predictor of variation in student achievement growth within the same model as primary teaching responsibility. This is understandable given the high correlation between the two variables, i.e., the presence of one washes out the impact of the other. Lastly, we fit a model that only incorporated student demographics as Level 1 predictors. The purpose of this was to establish the proportion of variation in student achievement growth that could be attributed to student demographics. This analysis (see Appendix E, Table E-19) showed that minority status, special education status, and free/reduced lunch status were all significant predictors of variability in student achievement growth. Overall, the amount of variability explained—15.6 percent—was significant. As might be expected, the coefficients were negative, indicating that minority students, special education students, and students receiving free/reduced lunch all tend to have lower student achievement growth than do their counterparts who are not minority students, not special education students, and not receiving free/reduced lunch. Summary The goal of this analysis was to determine the proportion of variance in student growth scores could be attributed to teacher level factors, specifically MKT scores, and student level demographics. Figure 11 displays the breakdown in the proportion of variance that can be explained by (1) teacher level factors, (2) primary teaching level, (3) MKT, (4) student demographics, and (5) other student level factors. Results show that primary teaching level accounts for only 4.7 percent of the variation in student achievement growth; MKT accounts for 0.2 percent of the variability in student achievement growth. Similarly, teaching responsibility and MKT were highly correlated, making it difficult


30

to discern the differential impact of these variables. Based on the analysis, though, it appears that variation in student achievement growth is more closely linked to teaching level than teacher MKT. At the same time, other teacher factors account for 10.7 percent of the variation in student achievement growth. Student level factors were the most impactful predictors of variation in student achievement growth. Within our sample, only 13.1 percent of the variability in student achievement could be attributed to the three demographic variables we used in our analysis—minority status, special education status, and free/reduced lunch status. Clearly, though, these variables far outweighed the impact of teacher-level differences in our analysis. Figure 11. Variation in Student Achievement Growth Explained

Overall, then, it is fair to say that while teacher MKT is important for student achievement growth, it does not outweigh the importance of student level variables that are outside the influence of the classroom teacher. Similarly, MKT is so closely linked to a teacher’s level of responsibility that it is difficult to disaggregate the impact of both factors on student achievement growth. What is clear is that growth appears to slow as students progress through the middle school grades thus teachers in those grades, who often have higher MKT than those in the elementary grades, demonstrate a negative relationship between MKT scores and student achievement growth. Again, this is not to diminish the importance of teacher MKT; it is just that within our sample, there are clearly other factors that are more significant predictors of student achievement growth. These results are consistent with results of other similar studies. As noted by Whitehurst (2002), roughly 20 percent of the differences in student achievement is associated with individual classroom (i.e., teachers) and the remaining 60 percent is associated with differences among children. Whitehurst notes, however, that student differences may also include the effects of prior year student achievement, something that our study did not consider, which may also be a reflection of teacher impact in prior years. Thus overall, it is safe to conclude that teachers do have an impact on student learning and that furthermore, as shown in our study, teacher content knowledge does make a difference.

13.1%71.4%

4.7%0.2%

10.7%Student Demographics

Other Student Factors

Teaching Level

MKT

Other Teacher Factors


31

Appendices


32

Appendix A. Phase II Research Questions

Area Inquiry Research Questions

Math Teacher Leader Models and School Impact

Value-Added Assessment of District-Wide Impact

To what extent can variability in student achievement on the WKCE be attributed to adoption of MMP principles at the school or classroom level? To what extent can variability in growth of student achievement on the district benchmark assessments be attributed to adoption of MMP principles at the school or classroom level?

Network Analysis of School Professional Communities for Mathematics

What relationships can be observed between school-based professional learning communities and the extent to which a school has adopted and implemented MMP principles and practices? What relationships exist between indicators of distributed leadership and different MTL models? What relationships can be observed between student achievement and different MTL models?

Transition to College Mathematics

Readiness for Post-Secondary Mathematics

What relationships can be observed between schools’ use of the MPS Math Readiness Test results and the impact on collaboration, counseling practices, curricular decisions, and instruction? What trends are evident in student performance on the MPS Math Readiness Test? What trends are observed in the proportion of MPS students taking three and four years of high school mathematics?

Placement into College Mathematics Courses

What trends are observed in the proportion of MPS and non-MPS graduates placed in remedial mathematics courses at MATC and UWM? What relationships can be observed among high school mathematics preparation (i.e., course enrollments, math units), performance on the MPS Post-secondary Math Readiness Test, and college placement test results?

Mathematical Knowledge for Teaching (MKT)

Impact of Mathematical Preparation on Pre-service Teachers’ MKT

Do prospective teachers with a mathematics minor have stronger MKT than those without? To what extent can variability in MKT be attributed to mathematical preparation (i.e., math foundation courses, math minor courses, methods courses, practicum experiences)? What is the relationship of MKT to other measures (e.g., Praxis scores) and to teaching practice?

Impact of Content Development on Math Teacher Leaders’ MKT

What is the impact of partnership-driven professional development on the mathematical knowledge for teaching of math teacher leaders? What is the relationship of teacher leader MKT to other measures (e.g., professional development hours, school math focus, student achievement) and to teacher leader practice?

Linking Teacher MKT to Student Achievement

What is the relationship between teacher MKT and student achievement? Is there a stronger relationship between teacher MKT and (a) student knowledge gains throughout the school year as measured by district benchmark exams or (b) student achievement as measured by the state standard test administered the subsequent fall?


33

Appendix B. Research Area 1 HLM Analysis

This section of the report summarizes what we have learned over the past year in our efforts to evaluate the MMP at the overall district level. In Phase II of the Milwaukee Mathematics Partnership there was more of a focus on the impact of the Math Teacher Leader at the school, and how that impact was affected by the different models used by different schools, in terms of increasing student achievement in mathematics. Methodology____________________________________________________________ Participants Everyone at the school and district level who had the potential to positively affect student achievement in mathematics was asked to respond to an online survey, hereafter referred to as the MMP Survey. This year we saw a slight decline in the number that responded to our online survey. The MMP survey was distributed from 2009 to 2011. In 2011, there were 1,937 MPS school employees who responded to the survey. Table B-1 identifies those that participated in the MMP Survey for the three years in which the survey was administered, in terms of their role in the MMP. Table B-1. Number of Respondents to the MMP Survey by Role in the MMP

Year 1 (2009) Year 2 (2010) Year 3 (2011) Frequency Percent Frequency Percent Frequency Percent

Math Teacher Leader 147 5.8% 126 6.4% 131 6.8%

Learning Team Member &

Mathematics Teacher 178 7.0% 281 14.3% 274 14.1%

LT Member (Administrative,

Literacy Coach, Etc.) 234 9.2% 260 13.2% 219 11.3%

Math Teacher Only 1347 53.2% 786 39.9% 842 43.5%

Other 624 24.7% 518 26.3% 471 24.3%

Total 2530 100% 1971 100% 1937 100%

Instrumentation The primary measures that were considered in the evaluation efforts this year were the MMP Survey administered in the spring of 2011, and the mathematics subtest of the Wisconsin


34

Knowledge and Concepts Exam (WKCE), administered to students in grades 3 through 8 in the fall of 2011. Responses to the MMP Survey were utilized to create variables that could differentiate schools in terms of the quality of their participation in, and implementation of, MMP related activities. Descriptions of each of the variables created from responses to survey items are depicted in Table B-2, along with the number of items that were utilized to create each variable and the overall reliability for each subscale obtained from the administration in the spring of 2011 of the MMP Survey. Table B-2. School Level Variables and Constructed from Survey Responses

Variable Description ni a

Math Focus How focused a school is on increasing student achievement in mathematics 8 0.92

Professional Development Quality Quality of professional development sessions attended 5 0.95

Consistency Consistency with which teachers use assessments to guide classroom practice 5 0.85

Engagement 1 How frequently teachers work to align curricula/use CABS 9 0.94

Engagement 2 How frequently teachers engage in activities designed to gauge progress using WKCE/Benchmark results 6 0.96

Engagement 3 How frequently teachers talk about the teaching/learning of math w/ other teachers 5 0.92

Teacher Assessment The degree of teacher comfort w/ mathematics knowledge 7 0.92

Student Metacognition The degree to which teachers feel students are dedicated to their own learning. 3 0.89

Alignment How aligned a school's curriculum is to standards and learning targets 5 0.95

Math Teacher Leader How supportive MTL is perceived to be in terms of helping increase student achievement in mathematics 9 0.98

Learning Team Quality

How supportive LT is perceived to be in terms of helping increase student achievement in mathematics 12 0.98

Math Teaching Specialist

How supportive MTS is perceived to be at a school in terms of helping increase student achievement in mathematics

5 0.95

The scores on these variables were then averaged across respondents within a school to come up with overall scores for individual schools. Table B-3 depicts the descriptive statistics for each of these variables from 2009 to 2011. As the table illustrates the average scores for each of these subscales were remarkably stable over the three years, with the exception of Engagement 1. Scores on this variable slightly increased each year. This implies that, in general, teachers in MPS are more likely to report working to align their curriculum to MPS standards and using CABS.


35

Table B-3. Descriptive Statistics for School Level Variables

Variables (Possible Scores)

Year 1 (2009) Year 2 (2010) Year 3 (2011)

N Mean SD N Mea

n SD N Mean SD

Math Focus (8 to 48) 142 31.51 2.48 113 31.31 2.68 149 31.43 2.29

Professional Development Quality (5 to 30) 142 24.38 2.25 113 24.21 2.87 149 24.55 2.65

Consistency (5 to 30) 142 19.81 2.17 113 19.83 2.82 148 20.26 2.45

Engagement 1 (9 to 54) 142 36.62 5.44 113 37.78 5.51 149 38.14 5.76

Engagement 2 (6 to 36) 142 21.30 3.43 113 22.72 3.61 149 22.36 3.69

Engagement 3 (5 to 30) 142 22.59 2.63 113 22.89 2.70 149 22.85 2.60

Teacher Assessment (7 to 42) 142 29.16 2.32 113 29.44 2.42 149 30.05 2.27

Student Metacognition (3 to 18) 142 12.36 1.95 113 12.67 1.80 149 12.54 1.44

Alignment (5 to 30) 141 25.76 2.02 113 25.87 1.81 149 25.96 1.98

Math Teacher Leader Quality (9 to 54) 141 46.39 6.03 113 46.41 5.21 149 46.35 5.61

Learning Team Quality (12 to 72) 142 55.39 7.17 113 55.27 8.65 148 56.05 6.77

Math Teaching Specialist Quality (5 to 30) 139 23.00 2.99 113 23.32 2.43 145 23.00 2.79

These school level variables were then used as independent variables in subsequent quantitative analyses undertaken to try and predict the student achievement in mathematics in the fall of 2011, as measured by the WKCE. A complete listing of all survey items on the MMP Survey that was administered in the spring of 2011, compiled by the particular variable each survey item was used to create, can be found in Appendix D. Results from the WKCE assessments were used as a measure of student achievement in mathematics. Scale scores were used for all of our analyses. This is a change from previous reports, in which MMP survey variables were used to predict MAP growth scores. In these earlier analyses, we found no statistically significant relationships between the MMP Survey and MAP growth. Given this fact, and the fact that the MMP Survey was not administered in the spring of 2012, this year we used MMP Survey variables obtained from the spring of 2011 to predict student achievement on the WKCE in the fall of 2011, in an attempt to evaluate the impact of MMP related activities on increasing student achievement in mathematics.


36

Analyses Only one of the three research areas explored in the evaluation of the MMP was related to increasing student achievement in mathematics, specifically the research area that focused on Hierarchical Linear Modeling (HLM), which was conducted in an attempt to answer the following research question:

To what extent can variability in student achievement on the WKCE be attributed to the adoption of MMP principles at the school level?

Results________________________________________________________________ Impact of the MMP on Increasing Student Achievement in Mathematics The first HLM model that was fit to the data utilized student WKCE math scale scores as the dependent variable. No school level predictors were included in the model, resulting in fitting an unconditional model, to determine the total amount of between school level variability that could be predicted by the inclusion of school level variables. These models were fit for two grade level bands: 3-5 and 6-8. The results of fitting these models are depicted in Tables B-4 and B-5. The variance components in the Table Ean be utilized to determine the intraclass correlation (ICC), which represents the proportion of variability in student level gain scores that exists between schools, and therefore can be explained by including school level variables in level 2 of the HLM model. The results of fitting the unconditional models are presented by corresponding grade level bands. Grades 3-5 For students in grades 3 - 5 the results of fitting an unconditional model are depicted in Table B-4. As Table B-4 illustrates, the 𝐼𝐶𝐶 = !"#.!"/(!"#.!"!!"#$.!") = .15. Therefore, approximately 15% of the variability in student level WKCE scores can be attributed to between school differences for students in grades 3 to 5. Table B-4. Results of Fitting Unconditional HLM Model for Grades 3-5 Fixed Effect Coefficient St. Error t Ratio df p-value WKCE (intercept) 432.29 2.22 194.95 100 < 0.001




Grades 6-8 For students in grades 6-8 the results of fitting an unconditional model are depicted in Table B-5. As Table B-5 illustrates, the 𝐼𝐶𝐶 = !"#.!"/(!"#.!"!!!"#.!") = .19. Therefore, approximately 19% of the variability in student level WKCE scores can be attributed to between school differences for students in grades 6 to 8.


37

Table B-5. Results of Fitting Unconditional HLM Model for Grades 6-8 Fixed Effect Coefficient St. Error t Ratio df p-value WKCE (intercept) 492.15 2.71 181.70 76 < 0.001




After fitting unconditional models to the data an exploratory analysis was conducted, for schools in each of these two grade level bands, to determine if any of the variables created from the MMP Survey could explain a statistically significant amount of between school variability. The results of the exploratory analysis indicated that Math Focus, Professional Development Quality, Teacher Assessment, Alignment, and Learning Team Quality were all statistically significant predictors of WKCE scores for students in grades 3-5 and grades 6-8. However, there was a very high correlation between Math Focus, which was found to be the strongest predictor, and each of other variables. Therefore, only Math Focus was used in the model to predict student WKCE score. Including any of the other predictors would have resulted in multicollinearity in the model. The results obtained from fitting HLM models, with Math Focus as a predictor, are depicted in Tables B-6 and B-7. As Table B-6 illustrates, for students in grades 3-5, schools that reported a greater focus on increasing student achievement in mathematics and talking more frequently about the teaching and learning of math with other teachers were more likely to have greater WKCE scores. Once again, the variance components from the two models fit can be used to calculate an ICC, which reflects how much variability is explained by including this Level 2 predictor in the model. Specifically, for grades 3 – 5, the 𝐼𝐶𝐶 = (476.17− 351.99)/476.17 =.26. Therefore, approximately 26% of the between school variability for schools that serve students in grades 3 - 5 can be explained by differences in how focused the school was on increasing students achievement in mathematics. Table B-6. Results of Fitting HLM Model with Math Focus as a Level 2 Predictor for Grades 3-5 Fixed Effect Coefficient St. Error t Ratio df p-value WKCE (intercept) 281.26 25.64 10.97 99 < 0.001 Math Focus 1.32 0.53 2.49 101 0.015



School Mean WKCE Scores 18.76 351.99 99 1863.64 <0.001 Level 1 Effect 52.05 2709.49

As Table B-7 illustrates, for students in grades 6-8, schools that reported having a greater focus on increasing student achievement in mathematics were more likely to have greater WKCE scores. Once again, the variance components from the two models fit can be used to calculate an ICC, which reflects how much variability is explained by including this Level 2 predictor in the model. Specifically, 𝐼𝐶𝐶 = (536.48− 436.87)/ 536.48 = .19. Therefore, approximately


38

19% of the between school variability for schools that serve students in grades 6 - 8 can be explained by differences in how focus on increasing students achievement in mathematics. Table B-7. Results of Fitting HLM Model with Math Focus as a Level 2 Predictor for Grades 6-8 Fixed Effect Coefficient St. Error t Ratio df p-value WKCE (intercept) 371.93 26.83 13.86 75 < 0.001 Math Focus 22.94 5.32 4.313 75 < 0.001



School Mean WKCE Scores 20.90 436.87 75 2164.98 <0.001 Level 1 Effect 47.10 2218.63


39

Appendix C. MMP Online Survey Questions and Constructed Variables

Variable Description Learning Team Quality

How supportive LT is perceived to be in terms of helping increase student achievement in mathematics

The LT helps our school focus on increasing student achievement in math. The LT is a valuable asset for increasing student achievement in math. The LT helps keep our school informed of district initiatives related to the teaching and learning of math. The LT helps our school focus on consistent teaching and learning of math. The LT sets clear expectations about how assessment is used to guide math teaching and learning in our school. The LT sets clear expectations that teachers within a grade band (i.e. K-5, 6-8, 9-12) use the same math textbook. The LT sets clear expectations that teachers use the District curriculum guides for math. The LT promotes math instructional coordination across grade levels in the school. The LT troubleshoots the implementation of school improvement efforts in math. The LT supports the implementation of school improvement efforts in math. The LT monitors the implementation of school improvement efforts in math. The LT provides direction for staff development programs for improving math teaching and learning. (Response Scale: Strongly Disagree—Disagree —Somewhat Disagree — Somewhat Agree — Agree —Strongly Agree)

Math Teacher Leader Quality

How supportive MTL is perceived to be in terms of helping increase student achievement in mathematics

The MTL is a valuable asset for improving instructional practice in the math classroom. The MTL keeps our school informed about district initiatives related to teaching and learning math. The MTL helps our school focus on increasing student achievement in math. The MTL helps our school focus on consistent teaching and learning of math. The MTL promotes math instructional coordination across grade levels in the school. The MTL provides leadership for the implementation of school improvement efforts in math. The MTL is actively involved in the implementation of school improvement efforts in math. The MTL facilitates staff development for improving instructional practice in the math classroom. The MTL engages teachers in professional learning experiences related to the


40

teaching and learning of math. (Response Scale: Strongly Disagree—Disagree —Somewhat Disagree — Somewhat Agree — Agree —Strongly Agree)

Engagement 1 How frequently teachers work to align curricula/use CABS

Promoting alignment between the assessments used from the school's math instructional program to learning targets. Promoting alignment between the math curriculum used and the math learning targets. Improving or adapting teacher-made math assessments to reflect the learning targets. Modifying math textbook assignments to better reflect the learning targets. Working on plans to improve the teaching of specific math learning targets identified as an area of need from the data. Examining results from CABS and discussing strategies to improve scores with other teachers. Examining and discussing examples of students' work in math. Using results from CABS to plan for improving student performance in math. Using samples of student work in planning for and evaluating school improvement activities. (Response Scale: Never—1 this year—few per year—1 per month—few per month—1-2 days per week or more)

Teacher Assessment The degree of teacher comfort w/ mathematics knowledge

Prior to teaching, teachers study and can articulate the math c1pts students will be learning. Teachers use student-friendly language to inform students about the math objective they are expected to learn during the lesson. Students can describe what mathematical ideas they are learning in the lesson. Teachers can articulate how the math lesson fits within the progression of student learning. Teachers use classroom assessments that yield accurate information about student learning of math concepts and skills and use of math processes. Teachers use assessment information to focus and guide teaching and motivate student learning. (Response Scale: Strongly Disagree—Disagree —Somewhat Disagree — Somewhat Agree — Agree —Strongly Agree)

Engagement 2 How frequently teachers engage in activities designed to gauge progress using WKCE/Benchmark results

Examine results from WKCE assessments in math and discuss strategies to improve scores. Examine results from district benchmark assessments in math and discuss strategies to improve scores. Establishing goals for school improvement in math based on WKCE and/or benchmark assessment data. Examining the school's overall progress toward improvement goals in math.


41

Using results from the school's WKCE/benchmarks math assessments to plan for improving student performance in math. Using the results from the WKCE and/or benchmark math assessments. (Response Scale: Never—1 this year—few per year—1 per month—few per month—1-2 days per week or more)

Math Focus How focused a school is on increasing student achievement in mathematics.

There is a strong focus on increasing student achievement at my school. There is a commitment at my school to improve student achievement in math. The leadership at my school helps us focus on student learning in math. There is a strong emphasis on improving math teaching and learning in my school. Teachers at my school are committed to improving math teaching and learning. Teachers at my school devote time for collaboration to improve math teaching and learning. (Response Scale: Strongly Disagree—Disagree —Somewhat Disagree — Somewhat Agree — Agree —Strongly Agree)

Math Teaching Specialist

How supportive MTS is perceived to be at a school in terms of helping increase student achievement in mathematics

The District MTSs provide important information that supports our school math improvement efforts. The District MTSs are knowledgeable about how to improve student achievement in math. The District MTSs provide important expertise that supports our school math improvement efforts. A District MTS has been actively engaged in facilitating professional development for improving math teaching and learning at my school. A District MTS has worked with me individually or small groups of teachers on math improvement activities during the past school year. (Response Scale: Strongly Disagree—Disagree —Somewhat Disagree — Somewhat Agree — Agree —Strongly Agree—Not Applicable)

Alignment How aligned a school's curriculum is to standards and learning targets

MPS learning targets. Wisconsin state standards. Comprehensive Math Framework. WKCE District Benchmark Assessments (Response Scale: Strongly Disagree—Disagree —Somewhat Disagree — Somewhat Agree — Agree —Strongly Agree)

Professional Development Quality

Quality of professional development sessions attended


42

Gave me many opportunities to work on aspects of my math teaching that I am trying to develop. Provided me with mathematical knowledge or information that is useful to me in the classroom. Allowed me to focus on a specific math' topic over an extended period of time. Focused my attention on particular things I was doing in the classroom during math instruction. Led me to think about an aspect of my math teaching in a new way. (Response Scale: Strongly Disagree—Disagree —Somewhat Disagree — Somewhat Agree — Agree —Strongly Agree)

Engagement 3 How frequently teachers talk about the teaching/learning of mathematics w/ other teachers

Provide feedback to a teacher who was trying to improve his/her instructional practices in math. Engage in conversations with teachers about the teaching and learning of math. Talk with teachers about the mathematical ideas that students should be learning. Talk with teachers about the mathematical c1pts that students are struggling to learn. Discuss students' reactions to a math lesson with teachers to improve classroom practice. (Response Scale: Never—1 this year—few per year—1 per month—few per month—1-2 days per week or more)

Consistency Consistency with which teachers use similar curricula and assessments to guide classroom practice

Within grade level bands (i.e. K-5, 6-8, 9-12), teachers use the same textbook. Within grade levels, teachers use the same classroom assessments (e.g., CABS) to evaluate student learning in math. Within grade levels, teachers use the same classroom assessments to guide math instruction. Within grade levels, teachers use the same classroom assessments to monitor student progress in math. (Response Scale: Strongly Disagree—Disagree —Somewhat Disagree — Somewhat Agree — Agree —Strongly Agree)

Student Metacognition The degree to which teachers feel students are dedicated to their own learning.

Students actively and regularly use descriptive feedback to improve the quality of their work. Students study the criteria by which their work will be evaluated by analyzing samples of strong and weak work. Students keep track of their own learning over time (e.g., journals, portfolios) and can articulate what areas need improvement. (Response Scale: Strongly Disagree—Disagree —Somewhat Disagree — Somewhat Agree — Agree —Strongly Agree)


43

Appendix D. Research Area 2 MATC Placement Data

Students who attend Milwaukee Area Technical College (MATC) who wish to pursue a degree there are, in many circumstances, required to take the Accuplacer Examination in mathematics. The examination may be waived if a student has (i) completed 12 or more credits in college academic courses (including math) with grades of C or better and a cumulative GPA of 2.0 or higher, or (ii) has submitted official ACT test scores with a composite score of 18 or higher. The purpose of this report is to establish baseline data on the results of this examination for recent graduates of both Milwaukee Public Schools (MPS) and non-MPS schools who apply for admission to MATC. Compared with data on student backgrounds for MPS graduates available from the University of Wisconsin-Milwaukee, the data we were able to collect from MATC turned out to be rather incomplete. We do not know the number of units of high school mathematics for any MATC students, and we generally do not have any ACT scores, so for the overwhelming majority of MATC students we only have one measurement of their mathematics knowledge prior to beginning their college work. What we do have, in contrast to the UWM data, is the outcome of the first college mathematics course at MATC for students who did take a mathematics course at MATC. Overall, and as we shall detail below, outcomes on the Accuplacer Arithmetic Test for all MATC students were poor, and the MPS students were, as a group, less well prepared than the counterparts from other secondary institutions. There were, however, no significant differences between MPS students and students attending other institutions in the City of Milwaukee. We have data for students completing high school in 2010, and applying to MATC for Fall 2010 admission. Note that many of these students never attended MATC. We have similar data for 2009. Between 2009 and 2010, however, MATC changed its course offerings and placement procedures. We will therefore discuss only the 2010 data in detail, appending the raw 2009 data in a final subsection. We have chosen to report data from high schools disaggregated into 4 categories. MPS Regular refers to MPS schools that are not charter schools or other non-traditional arrangements. Charter schools and similarly administered institutions partnered with MPS are listed as MPS Other. A separate sub-category of Non-MPS schools consists of other schools that are located in the City of Milwaukee. It is important to note that much of the data in this report is what statisticians would call self-reported in the sense that it was provided to MATC voluntarily. As a result, it is very incomplete and no effort should be made to use it to infer what is true about the MATC student population as a whole. For example, we have no way of knowing if the number of Math ACT scores reported is low because students did not report their scores, or because there were no scores to be reported. Similarly, since students are allowed to use a suitable ACT score in place of the Accuplacer, students with higher ACT scores may have opted out of the Accuplacer, depressing the Accuplacer scores. Finally, since MATC has a dual role as a technical school and a two-year liberal arts college, many students may forego mathematics placement testing, irrespective of


44

their mastery of the subject. All of this is in contrast to UWM where both the ACT and the UW System placement test are required of virtually all students entering directly from high school. Interpreting the Accuplacer Scores______________________________________________ The Accuplacer Mathematics Tests have three components: Arithmetic, Elementary Algebra, and College Level Mathematics. It should be noted that scores on each component range from 20 to 120, and do not directly reflect the number of correct responses as the test is given in a computer adaptive environment. For each test component, we have grouped scores in ranges, as found on the El Paso (Texas) Area College Readiness Initiative website (EEPC, 2004). Tables D-1 through D-3 summarize information on the interpretation of the scores from this resource. Table D-1. Interpretation of Accuplacer Arithmetic Scores Score Arithmetic skill level and specific skills 38-64 Minimal arithmetic skills. These students can: perform simple operations with whole

numbers and decimals (addition, subtraction, and multiplication); calculate an average, given integer values; solve simple word problems; identify data represented by simple graphs.

65-92 Basic arithmetic skills. These students can: perform the basic arithmetic operations of addition, subtraction, multiplication, and division using whole numbers, fractions, decimals, and mixed numbers; make conversions among fractions, decimals, and percents.

93-109 Adequate arithmetic skills. These students can: estimate products and squares of decimals and square roots of whole numbers and decimals; solve simple percent problems of the form p% of q = ? and ?% of q = r; divide whole numbers by decimals and fractions; solve simple word problems involving fractions, ratio, percent increase and decrease, and area.

110 or higher

Substantial arithmetic skills. These students can: find equivalent forms of fractions; estimate computations involving fractions; solve simple percent problems of the form p% of ? = r; solve word problems involving the manipulation of units of measurement; solve complex word problems involving percent, average, and proportional reasoning; find the square root of decimal numbers; solve simple number sentences involving a variable.

Table D-2. Interpretation of Accuplacer Elementary Algebra Scores Score Arithmetic skill level and specific skills 28-43 Minimal pre-algebra skills. These students demonstrate: a sense of order relationships and

the relative size of signed numbers; the ability to multiply a whole number by a binomial. 44-81 Minimal elementary algebra skills. These students can: perform operations with signed

numbers; combine like terms; multiply binomials; evaluate algebraic expressions. 82-108 Sufficient elementary algebra skills. These students can: add radicals, add algebraic

fractions, and evaluate algebraic expressions; factor quadratic expressions in the form ax2 + bx + c, where a = 1; factor the difference of squares; square binomials; solve linear equations with integer coefficients.

109 or higher

Substantial elementary algebra skills. These students can: simplify algebraic expressions; factor quadratic expressions where a = 1; solve quadratic equations; solve linear equations with fractional and literal coefficients and linear inequalities with integer coefficients; solve systems of equations; identify graphical properties of equations and inequalities.


45

Table D-3. Interpretation of Accuplacer College Level Mathematics Scores Score Arithmetic skill level and specific skills 39 or less

These students should take the Elementary Algebra test before any placement decisions are finalized.

40-62 These students can: identify common factors; factor binomials and trinomials; manipulate factors to simplify complex fractions. These students should be considered for placement into intermediate algebra.

63-85 These students can demonstrate the following additional skills: work with algebraic expressions involving real number exponents; factor polynomial expressions; simplify and perform arithmetic operations with rational expressions, including complex fractions; solve and graph linear equations and inequalities; solve absolute value equations; solve quadratic equations by factoring; graph simple parabolas; understand function notation, such as determining the value of a function for a specific number in the domain; a limited understanding of the concept of function on a more sophisticated level, such as determining the value of the composition of two functions; a rudimentary understanding of coordinate geometry and trigonometry. These students should be considered for placement into college algebra or a credit-bearing course immediately preceding calculus.

86-102 These students can demonstrate the following additional skills: understand polynomial functions; evaluate and simplify expressions involving functional notation, including composition of functions; solve simple equations involving trigonometric, logarithmic and exponential functions. These students can be considered for a Pre-calculus course or a non-rigorous course in beginning calculus.

103 or higher

Students scoring at this level can demonstrate the following additional skills: perform algebraic operations and solve equations with complex numbers; understand the relationship between exponents and logarithms and the rules that govern the manipulation of logarithms and exponents; understand trigonometric functions and their inverses; solve trigonometric equations; manipulate trigonometric identities; solve right-triangle problems; recognize graphic properties of functions such as absolute value, quadratic, and logarithmic. These students should be considered for placement into calculus.

2010 MATC Accuplacer Scores______________________________________________ In this subsection, we present Accuplacer results for the cohort of students completing high school in 2010, and applying to MATC for Fall 2010 admission. There were 3715 students in this cohort, but it should be noted that the students in our sample did not necessarily attend MATC. The tables show the raw numbers, and the bar charts show the percentages. Because of the significant number of students with no results, we have presented a second bar chart giving the breakdown for the students for which we have Accuplacer results. Arithmetic Based on the earlier interpretation of the scores, we will divide the students into 5 score groups: 20-37, 38-64, 65-92, 93-109, and 110-120. Realistically, students scoring below 93 are not ready for college level work that involves even the most rudimentary of mathematics skills.


46

Table D-4 shows the raw numbers on the Accuplacer Arithmetic segment, for students applying for admission to MATC in 2010. The first observation from Table D-4 is that we have Accuplacer scores for only 2186 students out of a total of 3715 (59%). Of those 2816, only 153 (7%, or 4% of all students) scored 93 or higher. Figures D-1 and D-2 show percentages for all students, and for students for whom we have Accuplacer results. Table D-4. 2010 MATC Accuplacer Arithmetic Scores All Students MPS Regular MPS Other Not MPS CITY, Not MPS No ACT, No ACC 1119 479 76 564 64 ACT, No ACC ARI 410 67 6 337 27 20- 37 1044 561 125 358 50 38- 64 583 204 46 333 24 65-92 406 99 21 286 8 93-109 117 31 5 81 4 110-120 36 4 1 31 2 Total 3715 1445 280 1990 179

Figure D-1. 2010 MATC Accuplacer Arithmetic Percentages, All Students

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

All Students

MPS Regular

MPS Other Not MPS CITY, Not MPS

110-‐120

93-‐109

65-‐92

38-‐ 64

20-‐ 37

ACT, No ACC ARI

No ACT, No ACC


47

Figure D-2. 2010 MATC Accuplacer Arithmetic Percentages, Students with Accuplacer Scores

Elementary Algebra Based on the EEPC interpretation for this test segment, we will again use 5 score categories: 20-28, 28-43, 44-81, 82-108 and 10-120. A score of 109 or higher on this segment corresponds roughly to the successful completion of MATH 095, and placement into MATH 105 or other credit-bearing courses, at UWM. MATC has multiple courses at the level of MATH 095 at UWM, including MATGEN 109 and MATGEN 110. Table D-5 shows the raw numbers on the Accuplacer Elementary Algebra segment, for students applying for admission to MATC in 2010. Figures D-3 and D-4 show percentages for all students, and for students for whom we have Accuplacer results. Table D-5. 2010 MATC Elementary Algebra Accuplacer Scores All Students MPS Regular MPS Other Not MPS CITY, Not MPS No ACT, No ACC 2708 1226 238 1244 132 ACT, No ACC ALG 452 81 13 358 32 20- 27 34 16 2 16 2 28- 43 135 47 10 78 0 44-81 292 61 16 215 10 82-108 87 14 1 72 3 109-120 7 0 0 7 0 Total 3715 1445 280 1990 179

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

All Students

MPS Regular

MPS Other Not MPS CITY, Not MPS

110-‐120

93-‐109

65-‐92

38-‐ 64

20-‐ 37


48

We have relatively little data on this segment. This is not surprising given the adaptive nature of the Accuplacer: if students score poorly on the Arithmetic portion they are not examined on Elementary Algebra. We have 555 results for Elementary Algebra, and 2186 results for Arithmetic, and all but 559 of those 2186 students scored in our bottom two categories on Arithmetic. Recall that a score of 109 or higher on the Elementary Algebra segment of the test should correspond to placement into a credit-bearing course at UWM. Only 7 of the 555 students for whom we have an Elementary Algebra score (1.3%) scored at this level, and none of those were from MPS. Since we know from Table 4 of Section 1 that 8% of MPS students were placed into credit-bearing courses at MATC in 2010, we hypothesize that relatively stronger students are using their ACT scores for placement at MATC, and choosing not to take the Accuplacer. We have no way at present of verifying this hypothesis, however. Figure D-3. 2010 MATC Accuplacer Elementary Algebra Percentages, All Students

Figure D-4. 2010 MATC Accuplacer Elementary Algebra Percentages, Students with Accuplacer Scores

0%

20%

40%

60%

80%

100%


No ACT, No ACC ACT, No ACC ALG 20-‐ 27

28-‐ 43 44-‐81 82-‐108

109-‐120

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%


20-‐ 27 28-‐ 43 44-‐81 82-‐108 109-‐120


49

Elementary Algebra Only 7% of the entire cohort of students under consideration reported any score in this category, and of those 260 students, all but 2 reported scores of 62 or lower. Given the adaptive nature of the test, and the results reported above for Arithmetic and Elementary Algebra this was to be anticipated, and we will not analyze this category as there is too little data to draw any other conclusions but this one. 2010 MATC Mathematics ACT Scores___________________________________________ From our 2010 cohort of 3715 students, 494 (13%) reported ACT mathematics scores. Since MATC does not require the ACT for admission, we do not know how many of the students in this study have taken the ACT. There is an incentive to report scores, as they may be used in place of the Accuplacer for placement decisions. In future years there should be a significant rise in the numbers of ACT Math scores reported by MPS students as well as now there is a district-wide policy that taking this test is a district expectation. An interpretation of ACT scores may be found in (Key & O’Malley, 2004), or at the ACT website, http://www.act.org/products/k-12-act-test. Table D-6 shows the raw numbers for students applying for admission to MATC in 2010; Figures D-5 and D-6 show the percentages for all students, and for those students reporting scores. Table D-6. 2010 MATC Mathematics ACT Scores All Students MPS Regular MPS Other Not MPS CITY, Not MPS No ACT 3221 1352 267 1602 146 1-12 3 1 0 2 1 13-15 33 15 3 15 3 16-19 249 59 10 180 13 20-23 133 13 0 120 14 24-27 64 5 0 59 2 28-32 11 0 0 11 0 33-36 1 0 0 1 0 Total 3715 1445 280 1990 179

Figure D-5. MATC Mathematics ACT Scores, All Students

0%

20%

40%

60%

80%

100%


No ACT 0-‐12 13-‐15 16-‐19 20-‐23 24-‐27 28-‐32 33-‐36


50

Figure D-6. MATC Mathematics ACT Scores, Students with Mathematics ACT Scores

We see that more students reporting scores have a score in the 16-19 range. According to the interpretation of ACT scores in (Key & O’Malley, 2004), students scoring in this range should be considered ready to take MATH 095 at UWM; thus, a student with a score of 19 or less would not be considered ready for a credit-bearing college mathematics course. Aggregating the data for MPS-Regular and MPS-Other in Table 7, we find that 88 out of 106 (83%) MPS students reporting ACT scores had scores of 19 or less, whereas 197 out of 388 (51%) of non-MPS students reporting ACT scores had scores of 19 or less. These results are broadly consistent with the data in Table 4 (93% and 62% remedial placement respectively), but suggest that students’ mathematics skills might be improving between the time they take the ACT and the time they apply for admission to MATC. 2010 MATC Course Results___________________________________________ We have the results for mathematics coursework for 837 students who enrolled in mathematics courses at MATC in the Fall of 2010: 615 students who enrolled in non-credit courses, and 222 who enrolled in credit-bearing courses. Success in this coursework is, at a minimum, an indication that these students are placed correctly and take their coursework seriously. MATC has several non-credit mathematics courses, and the distinction between them is not always clear. We will aggregate certain courses, which appear to contain elementary and middle school mathematics, under the labels MathB and MathPH respectively, and then consider the courses MatGen 109 and MatGen 110 separately. For comparison, MATGEN 109 and MATGEN 110 are each roughly equivalent to MATH 095 at UWM. No one credit-bearing course had sufficient enrollment to allow us to make any credible judgment as to student success, so we have grouped those courses together. Note that MATC does not use the grade of ‘F’, using “Unsatisfactory” in its place. MATC also uses the grade of ‘E’, indicating effort. This is neither a passing nor a failing grade. Finally, no

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%


0-‐12 13-‐15 16-‐19 20-‐23 24-‐27 28-‐32 33-‐36


51

explanation has been provided for why there are students listed as having enrolled in a course but for whom no grade is listed, given that a “W” indicates that a student has withdrawn from the course. Non-Credit Courses Tables D-7 and D-8 show the results for students enrolling in MathB (elementary school mathematics) and Math PH (middle school mathematics). Figures D-7 and D-8 show the respective percentages. Table D-7. 2010 MathB Results All Students MPS Regular MPS Other Not MPS A/A- 2 0 0 2 Pass 116 40 8 68 E 18 14 0 4 Unsatisfactory 16 10 1 5 Withdraw 55 27 4 24 No Grade 66 36 2 28 Total 273 127 15 131

Table D-8. 2010 MathPH Results All Students MPS Regular MPS Other Not MPS B+ 1 0 0 1 C 1 1 0 0 D 2 2 0 0 Pass 11 8 1 2 E 1 0 0 1 Unsatisfactory 1 0 0 1 Withdraw 5 3 0 2 No Grade 5 2 0 3 Total 27 16 1 10

Figure D-7. 2010 MathB Results as Percentages

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

All Students MPS Regular MPS Other Not MPS

A/A-‐ Pass E Unsatisfactory Withdraw No Grade


52

Figure D-8. 2010 MathPH Results as Percentages

There is not much data here to analyze, but it is clear from Table 8 that for this group of students, the MPS students who attended MPS high schools were not as well prepared, even for elementary school mathematics, as their counterparts who attended non-MPS high schools. To be specific, 48 out of 142 MPS students (34%) earned a passing grade in MathB, whereas 70 out of 131 non-MPS students (53%) did so. Tables D-9 and D-10 show the results for students enrolling in MATGEN 109 and MATGEN 110. Figures D-9 and D-10 show the percentages. Table D-9. 2010 MATGEN 109 Results All Students MPS Regular MPS Other Not MPS A/A- 46 11 1 34 B/B+ 46 12 2 32 C/C+/C- 33 6 1 26 D/D- 9 3 1 5 Unsatisfactory 14 7 0 7 Withdraw 34 13 4 17 No Grade 56 14 2 40 Total 238 66 11 161

Table D-10. 2010 MATGEN 110 Results All Students MPS Regular MPS Other Not MPS A/A- 12 0 0 12 B/B+ 13 2 0 11 C/C+/C- 14 1 0 12 D/D- 9 4 0 5 Unsatisfactory 4 1 0 3 Withdraw 9 4 2 3 No Grade 17 1 0 16 Total 77 13 2 62

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

All Students MPS Regular

MPS Other Not MPS

No Grade

Withdraw

Unsatisfactory

E

Pass

D

C

B+


53

Figure D-9. 2010 MATGEN 109 Results as Percentages

Figure D-10. 2010 MATGEN 110 Results as Percentages

Out of the 77 MPS students enrolled in MATGEN 109, 33 (43%) earned a grade of C- or better, whereas 92 out of 161 (57%) Non-MPS students did so. The number of students enrolled in MATGEN 110 is too small to draw any real conclusions, but it does appear from Table 11 that MPS students who enrolled in the course again received generally lower grades than non-MPS students. MATGEN 109 and 110 are roughly equivalent to Math 095 at UWM, where the success rates are a bit higher, but not radically so.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


A/A-‐ B/B+ C/C+/C-‐ D/D-‐ Unsatisfactory Withdraw No Grade

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%




54

Credit-Bearing Courses Table D-11 shows the results for students enrolling in credit-bearing courses, and Figure D-11 shows the percentages. Once again, the number of MPS students is too small to draw any meaningful conclusions. Table D-11. 2010 Credit-bearing Course Results All Students MPS Regular MPS Other Not MPS A/A- 21 4 0 17 B/B+ 38 4 0 34 C/C+/C- 37 0 0 37 D/D- 22 2 0 20 Unsatisfactory 20 4 0 16 Withdraw 30 8 2 20 No Grade 54 5 3 46 Total 222 27 5 190

Figure D-11. 2010 Credit-bearing Course Results as Percentages

2010 Delayed Entry Students___________________________________________ MATC enrolls many non-traditional students. This provides us with an additional pool of students whose academic progress we may examine. In this subsection, we will examine the progress of students who delayed entry by one year; that is, students who completed high school in 2009 and applied to MATC for admission in 2010. We have identified 1240 such students. For this analysis, we will only separate out the MPS Regular students, combining charter students with the Non-MPS group. Table D-12 shows the Accuplacer scores for our cohort of 1240 delayed-entry students, and Figure D-12 shows the percentages.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%




55

Table D-12. 2010 Delayed-entry Accuplacer Results All Students MPS Regular Not MPS No ACT, No ACC 278 93 185 ACT, No ACC ARI 104 7 97 20- 37 371 206 165 38- 64 257 114 143 65-92 158 40 118 93-109 58 12 46 110-120 14 1 13 Total 1240 473 767

Figure D-12. 2010 Delayed-entry Accuplacer Results as Percentages

We have 134 Math ACT scores for these students. Most of the scores are from students not attending MPS. Table D-13 shows the ACT scores for the delayed-entry cohort. Figure D-13 shows the score distribution as percentages of the entire cohort; Figure D-14 shows percentages of the 134 delayed-entry students reporting ACT scores. Table D-13. 2010 Delayed-entry Mathematics ACT Results All Students MPS Regular Not MPS No ACT 1106 459 647 13-15 6 1 5 16-19 63 9 54 20-23 39 4 35 24-27 21 0 21 28-32 5 0 5 Total 1240 473 767

0% 20% 40% 60% 80% 100%

All Students MPS Regular Not MPS

No ACT, No ACC ACT, No ACC ARI 20-‐ 37

38-‐ 64 65-‐92 93-‐109

110-‐120


56

Figure D-13. 2010 Delayed-entry Mathematics ACT Results as Percentages (All Students)

Figure D-14. 2010 Delayed-entry Mathematics ACT Results as Percentages (Students Reporting Mathematics ACT Scores)

By comparison with the students who did not delay entry by one year, these students are very poorly prepared, with the MPS students worse off. Only 52 of the 767 non-MPS students took a credit-bearing course, and only 16 of the 473 MPS students did so. As with the direct entry students, there is a glaring disparity between the two groups when it comes to presenting ACT scores. This disparity should disappear in future years, as pointed out earlier, since MPS has instituted new policies regarding the ACT. Table D-15 gives the grades earned by the students who took any mathematics course at all, and Figure D-16 shows percentages.

75%

80%

85%

90%

95%

100%


No ACT 13-‐15 16-‐19 20-‐23 24-‐27 28-‐32

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%


13-‐15 16-‐19 20-‐23 24-‐27 28-‐32


57

Table D-15. Delayed-entry Mathematics Course Results All Students MPS Regular Not MPS A/A- 18 1 17 B/B+/B- 24 4 20 C/C+/C- 29 8 21 D/D-/D+ 11 6 5 Pass 28 8 20 E 7 5 2 Unsatisfactory 20 15 5 Withdraw 55 22 33 No Grade 70 25 45 Total 262 94 168

Figure D-16. Delayed-entry Mathematics Course Results as Percentages

For these delayed-entry students, 71 out of 262 (27%) MPS students earned a grade of C- or better, whereas 58 out 168 (35%) of non-MPS students did so. These percentages are considerable lower than those of students who graduated high school in 2010. Neither group of delayed-entry students completed their mathematics courses successfully, with the MPS students faring worse than their non-MPS counterparts. Addendum: 2009 Direct Entry Students - MATC Accuplacer Scores_________________ Between Fall 2009 and Fall 2010, MATC changed its course offerings and placement procedures. Given this fact, and the incomplete nature of our data, we have chosen to use our 2010 data as the baseline for any future analyses, and have therefore not analyzed our 2009 data in any detail. The tables and charts below are included for completeness, and attempt to summarize the 2009 data in a parallel fashion to that from 2010, to the greatest extent possible.

0%

20%

40%

60%

80%

100%


A/A-‐ B/B+/B-‐ C/C+/C-‐

D/D-‐/D+ Pass E

Unsatisfactory Withdraw No Grade


58

Table D-16. 2009 MATC Accuplacer Arithmetic Scores All Students MPS Regular MPS Other Not MPS CITY, Not MPS No ACT, No ACC 1079 433 75 571 92 ACT, No ACC ARI 322 43 4 275 24 20- 37 1047 513 167 367 95 38- 64 583 208 47 328 72 65-92 412 93 18 301 56 93-109 157 33 3 121 17 110-120 40 6 2 31 4 Total 3640 1329 316 1994 360

Table D-17. 2009 MATC Accuplacer Arithmetic Scores as Percentages of All Students

Table D-18. 2009 MATC Accuplacer Arithmetic Scores as Percentages of Students with Arithmetic Scores

0%

20%

40%

60%

80%

100%


No ACT, No ACC ACT, No ACC ARI 20-‐ 37

38-‐ 64 65-‐92 93-‐109

110-‐120

0%

20%

40%

60%

80%

100%


20-‐ 37 38-‐ 64 65-‐92 93-‐109 110-‐120


59

Table D-17. 2009 MATC Accuplacer Elementary Algebra Scores All Students MPS Regular MPS Other Not MPS CITY, Not MPS No ACT, No ACC 2638 1113 271 1254 245 ACT, No ACC ALG 352 59 7 286 27 20- 27 68 20 18 30 14 28- 43 168 45 6 117 35 44-81 306 73 12 221 29 82-108 94 16 0 78 9 109-120 13 3 2 8 1 Total 3639 1329 316 1994 360

Figure D-19. 2009 MATC Accuplacer Elementary Algebra Scores as Percentages of All Students

Figure D-20. 2009 MATC Accuplacer Elementary Algebra Scores as Percentages of Students with Elementary Algebra Scores

0%

20%

40%

60%

80%

100%


No ACT, No ACC ACT, No ACC ALG 20-‐ 27

28-‐ 43 44-‐81 82-‐108

109-‐120

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%


20-‐ 27 28-‐ 43 44-‐81 82-‐108 109-‐120


60

Table D-18. 2009 MATC Mathematics ACT Results All Students MPS Regular MPS Other Not MPS CITY, Not MPS No ACT 3241 1263 305 1673 328 1-12 1 0 0 1 1 13-15 27 15 2 10 2 16-19 178 28 7 143 14 20-23 113 14 1 98 9 24-27 68 6 1 61 6 28-32 10 3 0 7 0 33-36 1 0 0 1 0 Total 3639 1329 316 1994 360

Figure D-21. 2009 MATC Mathematics ACT Results as Percentages of All Students

Figure D-22. 2009 MATC Mathematics ACT Results as Percentages of Students with Mathematics ACT Scores

75%

80%

85%

90%

95%

100%


No ACT 0-‐12 13-‐15 16-‐19 20-‐23 24-‐27 28-‐32 33-‐36

0%

20%

40%

60%

80%

100%


0-‐12 13-‐15 16-‐19 20-‐23 24-‐27 28-‐32 33-‐36


61

Table D-19. 2009 MATC MathB Results All Students MPS Regular MPS Other Not MPS Pass 121 46 9 66 E 27 12 9 6 Unsatisfactory 20 11 3 6 Withdraw 60 24 4 32 No Grade 40 16 5 19 Total 268 109 30 129

Figure D-23. 2009 MATC MathB Results as Percentages

Table D-20. 2009 MATC Math 100 Results All Students MPS Regular MPS Other Not MPS A/A- 34 2 0 32 B/B+ 49 8 3 38 C/C+/C- 43 11 2 30 D/D- 18 7 2 9 Unsatisfactory 18 6 0 12 Withdraw 56 20 4 32 No Grade 30 10 1 19

Figure D-24. 2009 MATC Math 100 Results as Percentages

0%

20%

40%

60%

80%

100%

All Students

MPS Regular

MPS Other

Not MPS

No Grade

Withdraw

Unsatisfactory

E

Pass

0% 20% 40% 60% 80% 100%


A/A-‐ B/B+ C/C+/C-‐

D/D-‐ Unsatisfactory Withdraw

No Grade


62

Table D-21. 2009 MATC Other 100 Level Course Results (Combined) All Students MPS Regular MPS Other Not MPS A/A- 36 4 0 32 B/B+ 43 3 3 37 C/C+/C- 46 5 2 39 D/D- 16 5 2 9 Unsatisfactory 13 1 0 12 Withdraw 31 2 4 25 No Grade 37 7 1 19 Total 212 27 12 173

Figure D-25. 2009 MATC Other 100 Level Course Results (Combined) as Percentages

Table D-22. 2009 MATC 200 Level and Above Course Results (Combined) All Students MPS Regular MPS Other Not MPS A/A- 10 1 0 9 B/B+ 24 4 2 18 C/C+/C- 22 3 1 18 D/D- 12 2 2 8 Unsatisfactory 15 1 2 12 Withdraw 26 5 3 18 No Grade 22 6 2 14 Total 131 22 12 97

Figure D-26. 2009 MATC 200 Level and Above Course Results (Combined) as Percentages

0%

50%

100%


A/A-‐ B/B+ C/C+/C-‐ D/D-‐

Unsatisfactory Withdraw No Grade

0%

50%

100%


A/A-‐ B/B+ C/C+/C-‐

D/D-‐ Unsatisfactory Withdraw

No Grade


63

Appendix E. Research Area 3 MKT Analysis

To study the impact on teachers’ mathematical knowledge for teaching (MKT), we are using measures from the Learning Mathematics for Teaching project at The University of Michigan. These items estimate ability in using mathematical knowledge in the context of teaching. The measures utilize item-response-theory (IRT) to evaluate items, construct scales, and report results. This third research area is comprised of three inquiries: (1) the impact of the UWM mathematical preparation program on the development of MKT; (2) the impact of mathematics content-based professional development on the MKT of mathematics teacher leaders; and (3) the relationship of teacher MKT to student achievement. Inquiry 3a: Impact of Mathematical Preparation on Pre-service Teachers’ MKT This inquiry addresses the mathematical preparation of elementary teachers and ways to measure the extent to which these preservice teachers are learning the content necessary for effective STEM teaching. The following questions are addressed in light of the results from a longitudinal study of the mathematical knowledge for teaching of preservice teachers throughout their teacher preparation program.

• Does the mathematical preparation of preservice elementary teachers impact their mathematical knowledge for teaching?

• What mathematics courses should be required of preservice elementary teachers electing a minor in mathematics?

• What differences exist in the content knowledge of preservice teachers electing a mathematics minor compared with other preservice teachers?

At the core of effective STEM teaching in the area of mathematics is deep knowledge of the content that is relevant to teaching and the habits of mathematical practice. Effective teachers are well aware of mathematical progressions and of the coherence of mathematics. Teachers with deep content knowledge can successfully help students understand complex ideas and make mathematical connections and applications, and help students develop mathematical proficiency. Content knowledge development begins in preservice teacher preparation with rigorous coursework aimed at ensuring that new teachers have the knowledge and skills needed to effectively begin their careers teaching mathematics.

The Milwaukee Mathematics Partnership (MMP), as one of its core initiatives, redesigned the mathematical preparation of teachers at the University of Wisconsin-Milwaukee. This work was based on recommendations highlighted in The Mathematical Education of Teachers (hereafter, MET) (CBMS, 2001) and Adding It Up (NRC, 2001). A vital aspect of the approach was the use of design teams comprised of a mathematician who was responsible for ensuring the course contained rigorous and correct mathematics; a mathematics educator who ensured the course aligned with current educational thinking and mathematics curricula; and a Teacher-in-


64

Residence, who ensured course material related to classroom practice. To that end, the MMP revised the foundational mathematics content courses required of all prospective elementary and early childhood preservice teachers. Then developed four new mathematics courses for preservice elementary teachers: (1) Mathematical Problem Solving, (2) Geometry, (3) Discrete Probability and Statistics, and (4) Algebraic Structures.

The development of the four new mathematics courses was of particular importance given that the university had just taken an extraordinary step to require all elementary education majors to choose a minor in mathematics or science, along with a second minor in social studies or English/language arts. In the past, when only one minor was required, most majors chose social studies and less than 8% chose mathematics. While the minor requirement had changed, the curriculum had not and education majors at first just elected from the existing array of mathematics courses. Thus, a goal of the MMP was to develop new courses specifically for the minor that built and deepened the content knowledge needed to teach mathematics in alignment with the MET and NRC recommendations and the findings of Ma (1999) and Ball and colleagues (Ball, 2003; Ball & Bass, 2003; Ball, Thames, & Phelps, (2008; Hill, Rowan, & Ball 2005). The 18-credit mathematics minor was in addition to two foundational mathematics courses for a total of 24 credits in mathematics content, in addition to six credits of mathematics methods. The minor also included existing coursework in calculus concepts. The four new courses are now permanent course offerings as part of the elementary education mathematics minor. The central question of interest is whether or not preservice teachers enrolled in these courses have, in fact, developed deep mathematics content knowledge necessary for effectively beginning their careers as STEM teachers.

To study the impact on preservice teachers’ mathematical knowledge for teaching (MKT), we used measures from the Learning Mathematics for Teaching (LMT) project at The University of Michigan (Hill, Ball, & Schilling, 2008; Hill, Sleep, Lewis, & Ball, 2007). These items estimate ability in using mathematical knowledge in the context of teaching. The measures utilize item-response-theory (IRT) to evaluate items, construct scales, and report results. We constructed our own scales from the bank of items and analyzed the data using a two-parameter model.

In our longitudinal study, we tracked math course enrollment and measured the MKT of preservice elementary teachers at four time points as they progressed through their teacher education program. The first MKT provided a baseline at the start of their mathematical foundation courses. The second MKT was measured after completion of the foundation courses and, most if not all, of the courses for the mathematics minor, but prior to taking the mathematics methods courses. The third MKT was taken at the beginning of the mathematical methods courses. The fourth MKT was completed at the conclusion of the mathematical methods courses. We were then able to compare the MKT of the elementary education major with a mathematics minor to those without a math minor.

We hypothesize that teachers electing a mathematics minor and who had taken the new mathematics content courses would demonstrate stronger MKT than those not electing the minor. This comparison group design allowed us to test this hypothesis with the goal of determining the effectiveness of the new content courses for deepening mathematical knowledge for teaching. In addition, we also tracked and measured the MKT of early childhood majors as an additional comparison group.


65

Sample_________________________________________________________________ The sample for this analysis focused on students who (1) had completed their pre-service education and (2) had taken the MKT assessment at all four points in time for either Number and Operations or Geometry. This produced a sample of 315 preservice teachers. Table E-1 below indicates the breakdown of this sample. Table E-1. Sample Breakdown by Program Area

Program Frequency Percent

MCEA Math 75 23.8

MCEA 101 32.1

Early Childhood 134 42.5

Other Program 5 1.6

Total 315 100.0

Program Progression_______________________________________________________ The first question of interest focuses on the courses taken by different groups of teachers throughout their preservice mathematics preparation. To assess this, we examined enrollment data for our sample of 315 preservice teachers, as well as ACT mathematics scores and Praxis II scores (this exam is taken at the end of the program prior to licensure). In examining enrollment data, we were interested in the proportion of our sample who had enrolled in non-credit mathematics courses (Math 90/95) as well as the proportion who had enrolled in mathematics focus courses (Math 275, 276, 277, and 278). Table E-2 displays these results.

Table E-2. Mathematics Course Enrollment by Program

Program

MCEA Math MCEA

Early Childhood Other Total

N 75 101 134 5 315 ACT Math Score 23.46 20.37 19.76 19.00 20.83 Math 090 0% 13% 14% 20% .10 Math 095 7% 31% 43% 60% .30 Math 175 92% 96% 96% 100% .95 Math 176 100% 100% 100% 100% 1.00 Math 275 91% 2% 0% 0% .22 Math 276 40% 0% 0% 0% .10 Math 277 99% 1% 0% 0% .24 Math 278 31% 0% 0% 0% .07 Praxis II Middle School Score 164.05 161.76 160.04 162.65


66

As shown in the above table, MCEA Math students stand apart from other preservice teachers in several ways. First, their average ACT Mathematics score is over 3 points higher than the scores for other MCEA students or Early Childhood students. Second, only a small proportion (7 percent) of MCEA Math students was required to enroll in non-credit mathematics courses compared with significantly higher proportions of other MCEA students (31 percent) and Early Childhood students (43 percent). This indicates that the overall mathematics competency of the MCEA Math students was higher upon entering college. Finally, and as expected, high proportions of MCEA Math students enrolled in the math focus courses compared with negligible or no enrollment by students in other programs. This is as intended given the focus courses were specifically designed with this group of students in mind. Descriptive Results_________________________________________________________ Tables C-3 and C-4 present descriptive statistics for the preservice teachers in our sample. Note that MCEA students took the MKT assessment at four points in time, while students electing the Early Childhood Education program took the assessment at three points in time. Table E-3. Number and Operations MKT Results

Program

Course Math 175 Pre-Test

Math 175 Post-Test

CURRINS 331 Pre-

Test

CURRINS 330/332

Post-Test MCEA Math Mean -.28 .05 .14 .46 Std. Deviation .63 .68 .64 .65 N 51 51 51 51 MCEA Mean -.65 -.21 -.31 -.02 Std. Deviation .74 .90 .62 .67 N 67 67 67 67 Early Childhood Mean -.72 -.50 -.38 Std. Deviation .63 .77 .60 N 99 99 99

Table E-4. Geometry MKT Results Course

Program

Math 175 Pre-Test

Math 175 Post-Test

CURRINS 331 Pre-

Test

CURRINS 330/332

Post-Test MCEA Math Mean -.11 .34 .37 .49 Std. Deviation .53 .77 .66 .70 N 60 60 60 60 MCEA Mean -.34 .07 -.10 -.02 Std. Deviation .60 .77 .65 .70 N 88 88 88 88 Early Childhood Mean -.44 -.10 -.22


67

Std. Deviation .50 .67 .52 N 103 103 103

The descriptive MKT results show that for each content area, MCEA Math minors started their program with higher MKT than other MCEA and Early Childhood preservice teachers. MCEA Math minors also made strong progress throughout their program, increasing their MKT in each content domain by the end of the required mathematical methods courses. It is also important to note that MCEA and Early Childhood preservice teachers also increased their MKT throughout their program showing strong overall improvement for number and operations and geometry. It is also interesting to note that smaller gains were made by the MCEA and Early Childhood preservice teachers following the foundation courses for number and operations and that the Early Childhood preservice teachers had a decrease in MKT for geometry. Repeated Measures Analysis_________________________________________________ From the descriptive results, we have asserted that MCEA math minors and other MCEA students made substantial gains in their MKT over the duration of their preservice teacher preparation. Our sample of preservice teachers contains only students who completed the MKT assessments for number and operations and geometry at each of four points in time—at the beginning and end of the mathematics foundation courses and at the beginning and end of the mathematics methods courses. Using this sample, we were interested in determining if MCEA math minors and other MCEA students made gains throughout their programs and if there were any differences in the gains demonstrated by these groups of students. The repeated measures analysis reported below used MKT assessment score as the within-subjects factor and MCEA program (math minor or non-math minor) as the between subjects factor. Figure E-1 below displays the descriptive results from this analysis for the number and operations MKT assessment. Note that both groups made gains but it appears that MCEA math minors made greater gains. What is most apparent are the gains made by MCEA math minors between the end of the foundation courses and the start of the methods courses when compared with the slight decline in MKT demonstrated by other MCEA students. Figure E-1: Number and Operations Overall Program Impact

175 Pre-‐Test 175 Post-‐Test 331 Pre-‐Test 332

Post-‐TestMCEA Math -‐.28 .05 .14 .46Other MCEA -‐.65 -‐.21 -‐.31 -‐.02All MCEA -‐.49 -‐.10 -‐.12 .19

-‐.80

-‐.60

-‐.40

-‐.20

.00

.20

.40

.60

MKT Score


68

Repeated measures analysis for number and operations indicated a within subjects effect (F=39.35, df=3, p=.000) but did not indicate an interaction effect between MCEA program and MKT scores. Analysis of between subjects effects showed that there was a difference in the level of MKT scores between MCEA math minors and other MCEA students (F=13.73, df=1, p=.000). This is clearly shown in Figure E-1. From this analysis, we conclude that (1) both groups of MCEA students made statistically significant gains in MKT scores throughout their program, (2) that MCEA math minors have higher levels of MKT, and (3) the gains demonstrated by each group of students were not statistically different from each other. Figure E-2 below displays the results for the geometry assessment. As with the number and operations results, MCEA math minors demonstrated higher MKT from the beginning all the way through their program. MCEA math minors also appeared to show greater gains throughout their program when compared with the other MCEA students. Also notable is the decline in MKT shown by these other MCEA students from the end of the foundation course to the beginning of the methods courses. Figure E-2: Geometry Overall Program Impact

Repeated measures analysis for geometry indicated a within subjects effect (F=29.67, df=3, p=.000) as well as an interaction between MCEA program and MKT scores (F=3.47, df=3, p=.016). Analysis of between subjects effects showed that there was a difference in the level of MKT scores between MCEA math minors and other MCEA students (F=16.26, df=1, p=.000). This is clearly shown in Figure 7. From this analysis, we conclude that (1) both groups of MCEA students made gains in MKT scores throughout their program, (2) that MCEA math minors have higher levels of MKT, and (3) the gains demonstrated by each group of students were statistically different from each other; i.e., program selection did make a difference in geometry MKT scores. Figure E-3 below displays results for the Early Childhood students for Number and Operations and Geometry. As shown, Early Childhood students showed gains in both their number and

176 Pre-‐Test 176 Post-‐Test 331 Pre-‐Test 332 Post-‐

TestMCEA Math -‐.11 .34 .37 .49Other MCEA -‐.34 .07 -‐.10 -‐.02All MCEA -‐.24 .18 .09 .19

-‐.40-‐.30-‐.20-‐.10.00.10.20.30.40.50.60

MKT Score


69

geometry content knowledge. Those gains were steady for the Number and Operations scale. For the Geometry scale, their knowledge increased by the end of the foundation courses and then decreased by the end of the methods course, suggesting a lesser focus on Geometry in the methods course. Figure E-3. Early Childhood Program Impact

Repeated measures analysis for Early Childhood students for Number and Operations indicated a within subjects effect (F=9.89, df=2, p=.000). Similarly, a within subjects effect was found for Geometry scores (F=13.08, df=2, p-.000). From this analysis, we conclude that Early Childhood student made significant content knowledge gains during their program. Impact of Mathematics Focus Courses__________________________________________ The results of the repeated measures analysis indicated that MCEA math minors made gains in MKT or held steady from the end of their foundation courses until the beginning of the mathematics method courses. This was not the case for other MCEA students who did not elect a math minor. These findings suggest that the additional mathematics preparation through the mathematics focus courses made a difference in preservice teacher MKT. There are four mathematics focus courses for preservice elementary teachers: (1) Math 275 Mathematical Problem Solving, (2) Math 277 Geometry, (3) Math 278 Discrete Probability and Statistics, and (4) Math 276 Algebraic Structures. Our approach was to use regression analysis to determine if enrollment in one or more of the math focus courses helped predict performance on the final MKT assessment for either the Number and Operations scale or the Geometry scale. Given the differences in MKT between MCEA Math minors, other MCEA students, and Early Childhood students, we would hypothesize that enrollment in the math focus courses did make a difference in MKT. Results of this analysis using the final Number and Operations MKT assessment score from Currins 330/332 as the dependent variable are presented below. In the analysis, we introduced the post-test score from Math 175, which focuses on number and operations based on the

175 Pre-‐Test 175 Post-‐Test 330 Post-‐TestNumber and Operations -‐.71 -‐.50 -‐.38

Geometry -‐.43 -‐.10 -‐.22

-‐1.00-‐.90-‐.80-‐.70-‐.60-‐.50-‐.40-‐.30-‐.20-‐.10.00

MKT Score


70

assumption that performance in that course would predict final performance on the MKT. This analysis revealed that the model significantly predicted Final Number and Operations MKT Assessment score (F(5,219) = 28.25, p=.000). R2 for the model was 0.39. Table E-5 displays the regression coefficients for each variable. Table E-5: Regression Coefficients Predicting Final Number & Operations MKT Assessment Score

Unstandardized Coefficients

Standardized Coefficients

B Std.

Error Beta t Sig. (Constant) -.087 .046 -1.901 .059 Math 175 Post Test .403 .047 .466 8.610 .000 Enrollment in 275 -.097 .243 -.056 -.401 .689 Enrollment in 276 -.051 .171 -.022 -.297 .767 Enrollment in 277 .757 .237 .446 3.194 .002 Enrollment in 278 -.348 .179 -.129 -1.944 .053

As shown in these results, Math 175 post-test score (t=8.61, p=.000) and Enrollment in Math 277 (t=3.19, p=.002) were significant predictors of Final Number and Operations MKT Assessment Score. This is an important finding that suggests enrollment in at least one of the math focus courses, Math 277, is an important predictor of preservice teacher MKT. Results of this analysis using the final Geometry MKT assessment score from Currins 330/332 as the dependent variable are presented below. In the analysis, we introduced the post-test score from Math 176 which includes study of geometry and measurement based on the assumption that performance in that course would predict final performance on the MKT. This analysis revealed that the model significantly predicted Final Geometry MKT Assessment score (F(5,252) = 29.46, p=.000). R2 for the model was 0.37. Table E-6 displays the regression coefficients for each variable. Table E-6: Regression Coefficients Predicting Final Geometry MKT Assessment Score

Unstandardized Coefficients

Standardized Coefficients

B Std.

Error Beta t Sig. (Constant) -.115 .039 -2.923 .004 Math 176 Post Test .441 .048 .479 9.228 .000 Enrollment in 275 .100 .206 .060 .483 .629 Enrollment in 276 .264 .154 .110 1.713 .088 Enrollment in 277 .280 .206 .172 1.361 .175 Enrollment in 278 -.103 .161 -.039 -.641 .522

As shown in these results, Math 176 post-test score (t=9.23, p=.000) was a significant predictor of Final Geometry MKT Assessment score. Enrollment in Math 276 was nearly significant (t=1.71, p=.088). This suggests that the most important predictor of final MKT Geometry


71

performance is performance in the Math 176 course, and that enrollment in the focus area courses does not contribute significantly to final performance. Conclusions___________________________________________________________________

Overall, preservice teachers are demonstrating gains in mathematical knowledge for teaching as measured by the MKT assessments administered throughout their program of study. Results indicated that MECA preservice teachers with a math minor have stronger MKT scores than those who did not elect a math minor.

For students who have completed their program, the elementary teachers with a math minor began and ended their program with higher MKT scores than the non-math elementary teachers and the early childhood teachers on measures of Number and Operations and Geometry. These results may be due to self-selection bias where individuals who are more successful in mathematics, generally, score better on the MKT assessments and then self-select a math minor. This assertion is supported by the higher ACT math scores of students electing a mathematics minor when compared with other students. An alternative explanation is that the additional math content courses taken by those electing a mathematics minor may explain the greater gains exhibited by these students. This appears to be true for the Number and Operations assessment where enrollment in Math 277 clearly predicted MKT assessment scores. This was not necessarily true, however, for the Geometry assessment where enrollment in math focus courses did not predict final Geometry assessment scores.

A final important question is how does the MKT of preservice teachers at the end of their program, compare with that of inservice teachers and even that of Math Teacher Leaders (MTLs) in the Milwaukee Public Schools. To evaluate this question, we used data collected over time from classroom teachers and MTLs and compared that to our sample of 315 program completers, broken out by academic program, used in the analysis throughout this report. Table E-7 displays the descriptive statistics for this analysis. Table E-7: Descriptive Statistics for MKT Comparisons

MKT Assessment

Program/Group

MCEA Math MCEA Early

Childhood Classroom Teachers

Math Teacher Leaders

Number and Operations

Mean .46 -.02 -.38 -.30 .50 SD .65 .67 .60 .76 .93 N 51 67 99 713 203

Geometry Mean .49 -.02 -.22 -.14 .32 SD .70 .70 .52 .80 .94 N 60 88 103 713 203

A one-way analysis of variance was conducted to determine if there were statistically significant differences in the MKT assessment scores between the five groups displayed above. For Number and Operations, the one-way ANOVA indicated significant differences across the five groups (F (4,1128)=52.77, p=.000). Post-hoc analysis revealed that MCEA Math Minors scored significantly higher than the other groups, with the exception of Math Teacher Leaders, with whom they scored similarly.


72

For Geometry, the one-way ANOVA indicated significant differences across the five groups (F(4,1162)=21.19, p=.000). Post-hoc analysis revealed that MCEA Math Minors scored significantly higher than the other groups, with the exception of Math Teacher Leaders, with whom they scored similarly.

This final analysis is important—it demonstrates that the mathematics preparation received by mathematics minors during their preservice teacher education compares favorably with the professional development received by Math Teacher Leaders throughout the MMP program. Similarly, preservice math minors can be expected to have stronger mathematics preparation than other preservice teachers as well as the typical classroom teacher in the Milwaukee Public Schools. All of this suggests that the MMP has been extremely successful in improving the level of preservice teacher preparation at the University of Wisconsin-Milwaukee.

Inquiry 3b: Impact of Content Development on Math Teacher Leaders’ MKT Important changes in the MTL program were made for the 2011-2012 school year. Most significantly, the number of MTLs supported by the school district fell from approximately 115 to fewer than 50. Some of these were tenured MTLs and others were new to the role. Each of these MTLs was assigned to support 3-4 different schools, rather than a single school. This shift in program design created a situation where some MTLs were new to the role given that tenured MTLs did not want to continue in the role given the new program design. At the same time, partnership-driven professional development of math teacher leaders (MTLs) continued during the 2011-12 school year. This development focused on improving MTL content knowledge in the domain of number and operations. Throughout the MMP program, MTLs have taken the MKT Assessment at various points in time. The first assessment for the Phase II work was administered in Fall 2008. Subsequent assessments were administered in the Spring of 2009, Spring 2010, Spring 2011, and Spring 2012. Each year there were several new MTLs. It was intended that these individuals also take a pre-assessment in the Fall of the year in which they first became an MTL. The objective of this inquiry was to evaluate MTL content knowledge gains as measured by the MKT assessment, as well as to detect any content knowledge differences between different groups of MTLs. Our approach to this analysis was to examine the differences between pre-assessments and post-assessments for a given year. Sample Our sample contained 300 unique MTLs with an average tenure of 3.43 years and a median tenure of 3 years (SD=2.20). Figure E-4 shows the breakdown of the sample by years as an MTL. As shown in Figure E-3, nearly half of the sample of MTLs (44 percent) had been an MTL two or fewer years. This suggests regular turnover of MTLs throughout the four-year MMP Phase II period.


73

Figure E-4. MTL Sample by Years as an MTL

Results Descriptive Analysis Table E-8 below displays the descriptive results from the MKT assessment for each year it was administered in our study. The comparison between baseline results and the results shown in the table are shown in the analysis below. It is important to note that not all MTLs in our sample of 300 took the MKT assessment each year because some MTLs may not have been in that role in a given year having either left or just started the role. Also note that the geometry MKT assessment was not administered in Spring 2010 and only the Number and Operations assessments was administered in 2012. Table E-8. Results of MTL MKT Assessments MKT Administration MKT Scale Spring 2009 Spring 2010 Spring 2011 Spring 2012 Number and Operations

Mean .33 .34 .36 -.09

Std. Deviation .84 .79 .85 1.06

N 137 125 118 39 Geometry Mean .49 -1.22 Std.

Deviation .81 .39

N 137 93 Algebra Mean .56 .65 .35 Std.

Deviation .75 .75 .61

N 137 125 118


74

These results show that MTL content knowledge is trending lower across the four years of the MMP Phase II program. A likely explanation for this is that the MTLs taking the assessments in more recent years are, on average, less tenured. For example, in Spring 2009, the 137 MTLs who took the Number and Operations assessment had an average tenure of 4.52 years. In Spring 2012, the average tenure of an MTL who took the MKT assessment was 4.10 years. Similarly, the highest average tenure for MTLs taking the MKT assessment occurred in Spring 2010 when MTLs had an average tenure of 4.83 years. This tenure not surprisingly coincides with high results on the Number and Operations and Algebra scales. Paired Sample Results An additional question of interest was the extent to which MTLs demonstrated knowledge gains between pre-assessments and post-assessments. Recall that an MTL took only one pre-assessment and then took one or multiple post-assessments depending on how many years the individual was an MTL. For example, an individual who was an MTL in from Fall 2008 through Spring 2011 would take the following assessments:

1. Fall 2008 Pre-assessment 2. Spring 2009 Post-assessment 3. Spring 2010 Post-assessment 4. Spring 2011 Post-assessment 5. Spring 2012 Post-assessment

Similarly, a new MTL in the 2011-2012 school year would take a pre-assessment in Fall 2011 and a post-assessment in Spring 2012. The following presents results of these analyses. Two separate analyses are displayed for each year. First, paired sample t-test results are shown. Second, the effect of ‘Years as an MTL’ as a covariate is indicated as significant or not. 2008-2009 Results Table E-9 presents the results of these analyses. Recall that MTLs took all three MKT assessment scales in 2008-2009—number and operations, geometry, and algebra. Table E-9. Paired Sample Results 2008-2009

Paired Sample Test Time as MTL

MKT Scale Statistic Pre-Test

Spring 2009

Mean Difference t df Sig. F Sig.

Number & Operations

Mean .30 .33 .03 .66 124 .51 1.06 .32 N 125 125 SD .82 .85

Geometry Mean .33 .47 .14 2.45 124 .02 .31 .59 N 125 125 SD .86 .82


75

Algebra Mean .44 .59 .15 2.68 124 .01 .27 .60 N 125 125 SD .68 .75

These results are positive and show that MTLs made significant gains on the Geometry and Algebra MKT scales, but not on the Number and Operations scale. Time as an MTL was not a significant predictor of mean score changes for any of the MKT subscales. Thus newer MTLs showed similar results to those MTLs who were more experienced at the time. 2009-2010 Results Table E-10 presents the results of this analysis for the 2009-2010 school year. The pre-assessments in this year were taken in Fall 2008 or in Fall 2009 if the individual was a new MTL. Thus, we should expect to find that years as an MTL does have an impact on MKT Assessment gains. Table E-10. Paired Sample Results 2009-2010



Spring 2010


Number & Operations

Mean .34 .36 .02 .41 104 .69 .68 .42 N 105 105 SD .80 .79

Algebra Mean .40 .65 .25 4.30 104 .00 1.55 .22 N 105 105 SD .69 .75

Again, MTLs demonstrated significant gains on the Algebra MKT assessment scale but not on the Number and Operations scale. Also, time as an MTL was not a significant predictor of MKT assessment gains. These results are consistent with the previous year. 2010-2011 Results Table E-11 presents results from the 2010-2011 analysis. Pre-tests for this sample were taken in either Fall 2008, Fall 2009, or Fall 2010. Table E-11. Paired Sample Results 2010-2011



Spring 2011


Number & Operations

Mean .25 .39 .14 2.25 100 .03 .51 .49 N 101 101 SD .83 .86

Geometry Mean .10 -1.24 -1.34 -12.15 82 .00 .39 .54 N 83 83 SD .89 .38


76

Algebra Mean .39 .33 -.06 .92 101 .36 .16 .69 N 102 102 SD .67 .58

These results are inconsistent with previous years results. First, MTLs demonstrated significant gains on the number and operations scale and demonstrated significant decline on the geometry scale. At the same time, there were no changes observed in the algebra results. Overall, this pattern of achievement contradicts previous results. However, as with previous years, time as an MTL was not a significant predictor of assessment scores. 2011-2012 Results Table E-12 displays results from the 2011-2012 school year. Recall that there were fewer MTLs this year and that the average tenure for an MTL in 2011-2012 was about 0.5 years lower than the average tenure in previous years, yet the median tenure was 4.0 years, which is higher than the median MTL tenure for the overall sample. Table E-12. Paired Sample Results 2011-2012



Spring 2012


Number & Operations

Mean .26 -.07 -.33 2.89 37 .01 176.7 .01 N 38 38 SD 1.01 1.07

Two important findings are evident from these results. First, for this group of MTLs, content knowledge declined from the pre-assessment to the post-assessment. Because MTLs may have taken the pre-assessment during a previous year (e.g., 2009 or 2010), this suggests that the professional development received by MTLs was not as effective during the 2011-2012 school year when compared with previous years. Second, this year is the only year when time as an MTL made a difference in achievement. This makes sense because of the change in MTL model, the average MTL tenure during this year was significantly lower than in previous years. Conclusions Partnership-driven professional development has been a significant component of the MMP since its inception. Each year, MTLs focused on a particular content knowledge strand and were tested to determine if their content knowledge improved. Overall, results suggest that MTLs made strong content knowledge gains in previous years, but that improvement has slowed, or even reversed, in recent years. For example, in the 2008-2009 school year, MTLs demonstrated improvement on the Geometry and Algebra scales of the MKT assessment. However, by Spring 2011, the overall level of Geometry knowledge demonstrated by MTLs was quite a bit lower than previously displayed.


77

The likely explanation for this is turnover in the MTL population. We know that in recent years, MTLs were less tenured than in previous years. In Spring 2009, the average tenure of an MTL in our sample was 4.5 years. By Spring 2012, the average tenure had fallen to 4.1 years. To test this conjecture, we examined the relationship between MTL tenure and MKT assessment results. Interestingly, we did not find a strong relationship between these two indicators suggesting that MTL tenure (i.e., accumulated professional development over many years), does not necessarily lead to stronger MKT. What then, might be the cause of changes in MKT assessment scores. First, the change in MTL model may have an impact. Most recently in 2011-2012, MTLs shifted from supporting a single school to supporting multiple schools. This may have detracted from professional development sessions and opportunities for MTLs. Second, there has been no measure of the quality of professional development delivered but MKT results suggest that it has degraded slightly as the Milwaukee Public Schools District has taken on greater responsibility for the professional development and University of Wisconsin-Milwaukee personnel have taken a lesser role. Despite these changes and inconsistent results, though, it is clear that the professional development has been impactful for many MTLs. The sample size alone—300 MTLs—suggest that impact has been broad and likely beneficial to many across the Milwaukee Public Schools District. Inquiry 3c: Linking Teacher MKT to Student Achievement Teacher MKT has been found to be related to student achievement (Hill, Rowan & Ball, 2005; Hill & Ball, 2004). Our work in Phase II is measuring teacher MKT for two purposes: (1) to evaluate the impact of continuing MTL and teacher professional development across MPS, and (2) to contribute to the body of knowledge about the relationship between teacher MKT and student achievement. Our questions are:

1. What is the relationship between teacher MKT and student achievement growth?

2. In addition to teacher MKT, what other teacher demographics might help explain variation in student achievement growth?

3. In addition to teacher demographics, what student demographics might help explain variation in student achievement growth?

Data for this study were compiled over a six-year period. Teacher MKT and demographic information were obtained from classroom teachers during the spring of each year from 2006 through 2011. The MKT assessment was scored and IRT ability estimates were computed for each MKT sub-scale—Number and Operations, Algebra, Geometry, and Probability and Statistics—and overall. The overall MKT assessment score and teacher demographics were used in these analyses as a teacher level predictor. Student achievement data were obtained from the Milwaukee Public Schools (MPS) through our partnership with the MPS Department of Research. We used data from the WKCE exam from


78

2005 through 2011. We calculated gain scores for individual students who took the WKCE during this period. To obtain a gain score, a student had to take the MKT in two consecutive years. Student demographics were also compiled from the MPS data. The critical challenge for this analysis was how to link teachers to students. For this analysis, we linked teachers to the students they had during the school year in which they took the MKT. For example, a teacher who took the MKT in Spring 2006 was linked to the students he or she had in class during the 2005-2006 school year. The corresponding student gain score used in the analysis was calculated by subtracting the Fall 2005 WKCE score from the Fall 2006 score. Similarly, a teacher taking the MKT in Spring 2011 was linked to students he or she had in class during the 2010-2011 school year and the corresponding student gain score was the difference between the student’s Fall 2010 and Fall 2011 student achievement scores. Using this approach allowed us to substantially boost the sample size for our analysis by incorporating multiple years of data into the analysis. Figure E-5 below illustrates this approach.

Figure E-5 Data Linking Approach

Once these links were established, Hierarchical Linear Modeling (HLM) was used to estimate the proportion of variation in student achievement gain scores that can be attributed to (1) teachers overall, (2) teacher MKT scores, and (3) other teacher demographic variables. HLM was used because student achievement results were nested within teacher MKT scores. HLM is the appropriate analysis to partition variance in a dependent variable (student achievement) according to multiple levels of factors (student level factors and teacher level factors). Descriptive Results The following presents the demographics of our samples of teachers and students. Our analysis incorporated 358 teachers linked to 8,356 students. Teacher Demographics Based on analysis from the 2010-2011 annual report, we found that there was one key demographic variable that predicted variability in student achievement growth. This was the level at which a teacher taught. This demographic was found to be highly correlated with overall MKT scores. Other demographics, such as Years Certified, Highest Degree Held, and


79

Professional Development Hours did not predict variability in student achievement growth and thus are not reported here. Table E-13 displays the relationship between Teaching Level and MKT Scores. MKT Scores were calculated as an aggregate of three scales—number and operations, algebra, and geometry. Note that even though 14 teachers indicated they were high school teaches, and that the WKCE is only administered in the 10th grade (thus there could be no growth score from 9th to 10th grade), these teachers were linked to student growth. It is likely that these teachers moved grade levels enabling them to be linked to students in our sample. Table E-13. Teacher MKT and Teaching Responsibility Primary Teaching Responsibility

Mean MKT

Std. Deviation N

% of Sample

Kindergarten to Grade 3 -.18 .67 112 31.3 Grades 4-5 -.08 .68 145 40.5 Grades 6-8 .21 .72 87 24.3 Grades 9-12 .75 .75 14 3.9 Total -.01 .72 358 100.0

Table E-13 indicates that across the entire sample of teachers, the average MKT score was essentially 0.0, which is consistent with the population average score. This confirms that our sample of teachers was typical and not overly biased in any particular way. Second, it is clear that as teaching level increases, so does MKT ability. This is consistent with our findings from Year 3 and suggests that the two variables—teaching level and MKT ability—co-vary. Thus in an HLM analysis, one but not both can be expected to predict student achievement growth. Student Demographics We were interested in three student demographics: (1) minority status, (2) special education status, and (3) free/reduced lunch status. Table E-14 below indicates the proportion of students in our sample of students who fall into each of these categories. While most students in the sample were unique, a student could have appeared in the sample more than once if he or she was the student of more than one of the unique teachers in our sample. Table E-14. Student Demographics

Student Demographic Yes No

N % N % Minority Student 5,106 61.1 3,250 38.9

Special Education Student 1,366 16.3 6,990 83.7 Free/Reduced Lunch 6,978 83.5 1,378 16.5

Within our sample of students, the proportions of special education students and free/reduced lunch students were consistent with the district overall. The proportion of minority students in our sample is lower than within the Milwaukee Public Schools: approximately 86 percent of students are African American, Hispanic, Asian, or Native American (MMP, 2011). Table E-15 below shows a comparison between student achievement growth on WKCE scale scores for each type of student.


80

Table E-15. Student WKCE Achievement Growth by Student Demographics Student Demographic Statistic Yes No Minority Student Mean 22.2 23.1

SD 33.3 32.1 N 5,106 3,250

Special Education Student Mean 18.5 23.3 SD 45.2 29.8 N 1,366 6,990

Free/Reduced Lunch Mean 22.4 23,3 SD 33.4 29.6 N 6,978 1,378

As shown in Table E-15, student achievement growth is consistent for minority and non-minority students as well as for students receiving free/reduced lunch and those not receiving this benefit. Students classified as special education students showed lower average growth than those not classified as special education. These results suggest that at minimum, special education status is likely to predict variation in student achievement growth. However, due to the large sample size, the other student demographics may also be predictors of variability in student achievement growth. Lastly, we were interested in the descriptive achievement of students of teachers in different grade bands. As shown in Table E-16, average gain score of students declined as a teacher grade level increased. This is consistent with the finding that teacher grade level and student growth are inversely related. Table E-16: Student WKCE Growth by Teacher Grade Level Primary Teaching Responsibility

Mean WKCE Gain Score

Std. Deviation N

Kindergarten to Grade 3 36.9 32.2 1,520 Grades 4-5 21.7 32.3 3,562 Grades 6-8 16.9 31.3 3,197 Grades 9-12 12.2 42.3 77 Total 22.5 32.8 8,356

HLM Analysis Hierarchical Linear Modeling (HLM) was used to identify the proportion of variation in student achievement growth that was due to (1) teacher level variables in general, (2) overall teacher MKT results, (3) other teacher demographics, and (4) student demographics. For these analyses, students were level 1 subjects and teachers were level 2 subjects, i.e., students were nested within teachers. We were able to match 8,356 of the students in our sample to 358 different teachers across the study period.


81

Student proficiency growth was the outcome variable of interest. Proficiency growth was defined as the difference in WKCE scale score in two consecutive years. The average gain score growth for our sample of students was 24.47 scale score points (SD=32.96) for all students in our sample across all years. Several models were fit to these data. First, an unconditional model was fit to determine how much variability in the student achievement growth could be attributed to between teacher variability. If the unconditional model was non-significant, there was no value in continuing the analysis since that would mean that between teacher differences did not account for variability in student outcomes. Table E-17 depicts the results from fitting the unconditional model to the sample of students using student scale score growth as the dependent variable. These results indicate that 15.5 percent (171/(171 + 932)) of the variability in student scale score growth can be attributed to teacher level variables. Thus, it is worthwhile to fit a conditional model with teacher MKT and other demographics as predictor variables. Table E-17. Results of Fitting the Unconditional Model

Estimated Fixed Effects Coefficient SE t-ratio df p

Mean student scale score growth 25.4 0.80 31.7 357 .000

Estimated Random Effects SD Var df χ2 p Mean student scale score growth within teachers 13.1 170.7 357 1780.9 .000

Student scale score gain 30.5 931.9 At the same time, we conducted an exploratory analysis using Level 2 (teacher) variables as potential predictors of variation in achievement growth. Table E-18 indicates that potential Level 2 predictors were primary teaching responsibility (i.e., teaching level) and MKT assessment scores. Table E-18. Potential Level 2 Predictors

Predictor Coefficient SE t-ratio

Primary teaching responsibility -6.37 0.62 -10.29

MKT assessment score -2.31 0.82 -2.82 Note. These results are interpreted by examining the t-ratio for statistical significance. A value in excess of 1.76 indicates significance at the .05 level. Once it was determined that 15.5 percent of the variability in student outcomes was due to between teacher differences, a conditional model was fit that introduced primary teaching level and MKT assessment results at the teacher level to determine if they help explain a statistically significant proportion of the variability in student achievement growth. Table 6 displays the results from fitting a conditional model in which primary teaching level and overall teacher MKT score was used to attempt to explain between teacher variability.


82

The results of fitting the conditional model (Table E-19) indicate that primary teaching responsibility does help explain additional variation in WKCE scale score growth year over year. The amount of variability explained is significant—31.2 percent—((170-117)/170). Also, the coefficient is negative indicating that primary teaching responsibility is a negative predictor. We know from Table E-13 that teachers teaching at higher grade levels demonstrate stronger MKT assessment scores. It stands to reason, then, that student growth must slow at higher grade levels (for example from Grade 6 to Grade 7, or Grade 7 to Grade 8). Thus the relationship between primary teaching responsibility and student achievement growth is negative. Table E-19. Results of Fitting the Conditional Model using Teacher MKT Scores and Primary Teaching Responsibility as Predictors of Student Scale Score Growth


Mean student scale score growth 53.9 2.83 19.06 355 .000

Primary Teaching Responsibility -9.52 0.91 -10.45 355 .000

MKT Assessment Score -0.71 1.02 -0.70 355 .484

Estimated Random Effects SD Var df χ2 P Mean student scale score growth within teachers 10.79 117 355 1315.6 .000

Student scale score gain 30.51 931 At the same time, teacher MKT was not a significant predictor of variation in student achievement growth within the same model as primary teaching responsibility. This is understandable given the high correlation between the two variables, i.e., the presence of one washes out the impact of the other. When we fit a model using only teacher MKT as a Level 2 predictor, we found that MKT score was a significant predictor of variability in student achievement growth. As shown in Table E-20, MKT score is also a negative predictor of student achievement growth which is consistent with the results shown in Table E-19, reconfirming that MKT and Teaching Level are highly correlated indicators of student achievement growth. Table E-20. Results of Fitting a Conditional Model with MKT Score as a Predictor of Student Achievement Growth Estimated Fixed Effects Coefficient SE t-ratio df p

Mean student scale score growth 25.40 .79 31.8 356 .000

MKT Assessment Score -3.25 1.13 -2.8 356 .005

Estimated Random Effects SD Var df χ2 P

Mean student scale score growth within teachers 12.96 168 356 1768.9 .000

Student scale score growth 30.52 931 Based on this analysis, the amount of variability explained, while statistically significant, is actually quite minimal—1 percent—((170-168)/170.


83

Lastly, we fit a model that only incorporated student demographics as Level 1 predictors. The purpose of this was to establish the proportion of variation in student achievement growth that could be attributed to student demographics. This was akin to conducting a simple regression analysis. Table E-21 displays these results. Table E-21. Results of Fitting a Model with Student Demographics as Level 1 Predictors


Mean student scale score growth 25.52 .81 31.5 357 .000

Minority Status -6.02 1.37 -4.4 8351 .000

Special Education Status -15.82 .94 -16.9 8351 .000

Free/Reduced Lunch Status -3.05 .93 -3.29 8351 .001

Estimated Random Effects SD Var df χ2 P Mean student scale score growth within teachers 13.55 184 357 2114.6 .000

Student scale score growth 28.0 786 These results show that minority status, special education status, and free/reduced lunch status are all significant predictors of variability in student achievement growth. Overall, the amount of variability explained—15.6 percent—is significant (931-786/931). Also note that the coefficients are negative indicating that minority students, special education students, and students receiving free/reduced lunch all tend to have lower student achievement growth than do their counterparts who are not minority students, not special education students, and not receiving free/reduced lunch. Conclusions_____________________________________________________________ The goal of this analysis was to determine the proportion of variance in student growth scores could be attributed to teacher level factors, specifically MKT scores, and student level demographics. Figure E-6 below displays the breakdown in the proportion of variance that can be explained by (1) teacher level factors, (2) primary teaching level, (3) MKT, (4) student demographics, and (5) other student level factors. Results show that primary teaching level accounts for only 4.7 percent of the variation in student achievement growth; MKT accounts for 0.2 percent of the variability in student achievement growth. Similarly, teaching responsibility and MKT were highly correlated, making it difficult to discern the differential impact of these variables. Based on the analysis, though, it appears that variation in student achievement growth is more closely linked to teaching level than teacher MKT. At the same time, other teacher factors account for 10.7 percent of the variation in student achievement growth. Student level factors were the most impactful predictors of variation in student achievement growth. Within our sample, only 13.1 percent of the variability in student achievement could be attributed to the three demographic variables we used in our analysis—minority status, special


84

education status, and free/reduced lunch status. Clearly, though, these variables far outweighed the impact of teacher-level differences in our analysis. Figure E-6. Variation in Student Achievement Growth Explained

Overall, then, it is fair to say that while teacher MKT is important for student achievement growth, it does not outweigh the importance of student level variables that are outside the influence of the classroom teacher. Similarly, MKT is so closely linked to a teacher’s level of responsibility that it is difficult to disaggregate the impact of both factors on student achievement growth. What is clear is that growth appears to slow as students progress through the middle school grades thus teachers in those grades, who often have higher MKT than those in the elementary grades, demonstrate a negative relationship between MKT scores and student achievement growth. Again, this is not to diminish the importance of teacher MKT; it is just that within our sample, there are clearly other factors that are more significant predictors of student achievement growth. These results are also consistent with results of other similar studies. As noted by Whitehurst (2002), roughly 20 percent of the differences in student achievement is associated with individual classroom (i.e., teachers) and the remaining 60 percent is associated with differences among children. Whitehurst notes, however, that student differences may also include the effects of prior year student achievement, something that our study did not consider, which may also be a reflection of teacher impact in prior years. Thus overall, it is safe to conclude that teachers do have an impact on student learning and that furthermore, as shown in our study, teacher content knowledge does make a difference.

13.1%71.4%

4.7%0.2%

10.7%Student Demographics

Other Student Factors

Teaching Level

MKT

Other Teacher Factors


85

Appendix F. References

Borsuk, A. J. (2007, May 14). MPS' ideas bring unity: Superintendent, union leader join forces to advocate lofty learning goals. Milwaukee Journal Sentinel. Retrieved November 21, 2010, from http://www.jsonline.com/story/index.aspx?id=605204

Ball, D. L. (2003). What mathematical knowledge is needed for teaching mathematics? Paper presented at the U.S. Department of Education Secretary’s Mathematics Summit (February 6, 2003). Retrieved May 1, 2006 from http://www.ed.gov/inits/mathscience/ball.html.

Ball, D. L., & Bass, H. (2003). Toward a practice-based theory of mathematical knowledge for teaching. In B. Davis & E. Simmt (Eds.), Proceedings of the 2002 Annual Meeting of the Canadian Mathematics Education Study Group, (pp. 3-14). Edmonton, AB: CMESG/GCEDM.

Ball, D.L., Thames, M.H., & Phelps, G. (2008). Content knowledge for teaching: What makes it special? Journal of Teacher Education, 59, 389-407.

Conference Board of the Mathematical Sciences. (2001). The mathematical education of teachers. Providence, RI: American Mathematical Society and Mathematical Association of America.

Hill, H. C. & Ball, D. L. (2004). Learning mathematics for teaching: Results from California’s Mathematics Professional Development Institutes. Journal of Research in Mathematics Education 35, 330-351.

Hill, H. C., Rowan, B., & Ball, D. L. (2005). Effects of teachers’ mathematical knowledge for teaching on student achievement. American Educational Research Journal (42), 371-406.

Hill, H., Ball, D. L., & Schilling, S. (2008). Unpacking “pedagogical content knowledge”: Conceptualizing and measuring teachers’ topic-specific knowledge of students. Journal for Research in Mathematics Education, 39, 372-400.

Hill, H. C., Sleep, L., Lewis, J. M., & Ball, D. L. (2007). Assessing teachers' mathematical knowledge: What knowledge matters and what evidence counts? In F. K. Lester (Ed.), Second handbook of research on mathematics teaching and learning (pp. 111-155). Charlotte, NC: Information Age Publishing.

Huinker, D., Kranendonk, H., McLeod, K., & Farley, K. (2011). Milwaukee Mathematics Partnership Final Report. Milwaukee, WI: University of Wisconsin-Milwaukee.

Key, E., & O’Malley, R. (2004). Effect of four years of high school mathematics on placement testing at UWM. Retrieved November 21, 2010, from http://www4.uwm.edu/Org/mmp/PDFs/fouryear copy3.pdf

Ma, L. (1999). Knowing and teaching mathematics: Teachers’ understanding of fundamental mathematics in China and the United States. Mahwah, NJ: Lawrence Erlbaum.

Milwaukee Public Schools. (2007) Action Plan to Improve Milwaukee Public Schools 2007-2012. Milwaukee, WI: Author

National Research Council. (2001). Adding it up: Helping children learn mathematics. Mathematics Learning Study Committee, Center for Education, Division of Behavioral Sciences and Education, National Research Council. Washington, DC: National Academy Press.

University of Wisconsin System. (2010) Contents of the Mathematics Placement Test. Retrieved November 21, 2010, from http://testing.wisc.edu/math test.html.

Whitehurst, G. J. (2002) The Whitehouse Conference on Preparing Quality Teachers. Washington DC: The Whitehouse.

Documents

Milwaukee)Mathematics)Partnership) PhaseII · PDF fileMilwaukee)Mathematics)Partnership) ... This section includes a summary of this year’s work, with additional data and results