State Policies for Science and Mathematics Assessment:
Implementation of STEM K-12 Education Indicators
An Online Reporting System developed by NORC at the University of Chicago With support from the National Science Foundation
Year-One Results: October 2016
Developed under NSF Grant # 1544123
Rolf K. Blank, Principal Investigator
NORC at University of Chicago
Education and Child Development Department
mailto:[email protected]
NORC | State Policies for Science and Mathematics Education: Year One Results
1
A System for Reporting State Science and Mathematics Assessment Policies
NORC at the University of Chicago is developing and implementing an online reporting system
on state assessment policies with the goal of establishing one of the STEM K-12 education
indicators recommended by a National Research Council committee of leading science and
mathematics educators and researchers (Monitoring Progress toward Successful K-12 STEM
Education: A Nation Advancing? (NRC, 2013). This first-year report summarizes the progress of
implementation and initial findings from analysis of state policies across 20 states that
participated in the project in 2015-16 school year.
The NORC plan for developing and implementing the state assessment policies reporting system
has three key objectives:
1. With cooperation of the state departments of education, NORC will design and implement an online reporting instrument that will address the need for 50-state
information on the types of student assessments, alignment of assessments to content
standards, and state initiatives to develop approaches for science and math assessment.
2. Information will be collected across states using existing state websites and an online survey system to support cooperation with state education staff for annual updates on
policy changes and assessment initiatives.
3. NORC will design and implement a central web-based reporting resource that is accessible and useful to educators, leaders, and researchers, and NORC will provide
summary analysis of trends in assessment policies across the states.
The progress report is organized for access and use through the NORC project website. Each of
the main sections on assessment policies are reported through summary tables with state-by-state
information. This initial report demonstrates how information on state policies across the states
will be reported using the policy information for the 2015-16 school year from 20 states that
voluntarily participated in the first year of the project.
The selection and organization of assessment policy measures was based on results of a design
pilot study conducted in 2014-15 and through review and input from the project technical
advisers comprised of education researchers and science and mathematics education specialists.
Each of the sections of the first-year progress report provide a brief initial analysis of the cross-
state patterns in state assessment policies, and these initial results are reported with the
http://www.nap.edu/catalog.php?record_id=13509http://www.nap.edu/catalog.php?record_id=13509
NORC | State Policies for Science and Mathematics Education: Year One Results
2
understanding that more complete and meaningful analysis will follow in years two and three as
more states are participating in the system.
State Science and Mathematics Assessments: Which Assessments? Intended Uses?
What Types of Assessment Items and Tasks?
Assessment Administration and Reporting: What are State Policies?
Content of Mathematics and Science Assessments: What is the degree of alignment to
Standards?
Rationale for a STEM K-12 Indicator on State Assessment Policies
Summary: Year-One Results and Recommendations
NORC | State Policies for Science and Mathematics Education: Year One Results
3
State Science and Mathematics Assessments: Which Assessments? Intended Uses?
For the first year of the state assessment polices reporting system, the 2015-16 school year,
NORC requested information on the types of student assessments in mathematics and science
education that are funded and provided under state policy. Questions about state policies were
organized under two kinds of state policies for student assessments. First, states reported on
state-administered assessments that were required of most students in mathematics and science
(except students with disabilities taking alternate assessments). The information was reported
for grades 3-8 and high school assessments. Second, NORC requested information on
assessments that were supported by the state but were not required – the assessments were
developed as an option for local district or school decision on application. The state-supported
assessments might be assessments at the K-2 grade levels, diagnostic assessments, formative
assessments, benchmark assessments, or high school level assessments supported as an option,
including end of course, AP exams, or college or career readiness tests such as ACT, or SAT.
A key policy question for math assessments that was addressed with the NORC reporting system
was the role of consortia, i.e. whether assessments in mathematics were adopted by state policy
from a state assessment consortium (either Smarter Balanced or PARCC), or assessments were
adopted for the specific state. A second major question addressed by the NORC reporting
system was the intended uses of the assessment information, and several categories of uses were
outlined, based input from the project technical advisers including state education specialists.
The intended uses were reported for each of the state-administered and state-supported math and
science assessments.
Elementary & Middle Mathematics Assessments
The assessments listed in Table 3 for elementary and middle grades (grades 3-8) math show that
nine of the 20 states reporting for 2015-16 used assessments from the Smarter Balanced
Consortium, and two states used assessments in mathematics from the PARC Consortium. The
other nine states had elementary math assessments that were developed specifically for their
state. (Note: The NORC survey did not include information on assessment contractors by state.)
The format for reporting on state assessments and intended uses is shown below.
NORC | State Policies for Science and Mathematics Education: Year One Results
4
State Name of Assessments
Used for School
Accountability?
Used for Teacher
Accountability?
Used for Student
Accountability?
Used to Inform
Instruction/ Curriculum?
Arkansas ACT Aspire Yes Yes Yes Yes
Connecticut Connecticut Smarter Balanced Assessments
Yes No No Yes
District of Columbia
Partnership for Assessment of Readiness for College and Careers (PARCC)
Yes No No Yes
Delaware Smarter Mathematics (SBAC)
Yes Yes Missing Yes
Idaho
Idaho Standards Achievement Test (ISAT) by Smarter Balanced
No Yes No Yes
Kentucky
Kentucky Performance Rating for Educational Progress (K-PREP) (Vendor-Pearson)
Yes No No Yes
Stanford 10 (Norm Reference Test)
No No No No
Insert Table 3 followed by Examples of C/I uses in Table 4
Of the 20 states reporting on policies for 2015-16, 18 states reported an intended use of the
grades 3-8 math assessments for school accountability, 12 states reported they were to be used
for teacher accountability, four states reported a use for student accountability, and 16 states
reported the assessment results would inform curriculum and instruction in mathematics. State
education staff were asked to provide an example of how the math assessment results would be
used by educators in their state, and the most common use cited was that schools use the results
to make decisions about curriculum and how to adjust instrument to better meet needs.
NORC | State Policies for Science and Mathematics Education: Year One Results
5
Definitions: Intended Uses of Assessments in Mathematics and Science
1 Accountability
Student level – determine whether students advance to the next grade level, receive course credit,
or receive a diploma
Teacher level – determine how teachers are categorized into a teaching performance category or
whether teachers receive a bonus
School level – determine whether schools are meeting annual goals or are placed on an
improvement plan
2 Inform instruction or curriculum – provide feedback about instructional practice, guide
curriculum decisions, or determine appropriate instructional intervention for students
3 Student placement – determine which students are placed in a specific instructional program,
grade-level, or curriculum
4 College readiness – determine whether students are prepared for college level work
5 Career readiness -- determine whether students are prepared for jobs or careers
6 Graduation requirement – Passing assessment required for high school graduation (may be one
or more EOC assessments)
For grades 3-8, several states reported their agency had a role in supporting additional
assessments. For example, nine states reported they support interim benchmark assessments
(including states in Smarter Balanced consortia), several states provide diagnostic assessments
(e.g., MN, PA, WV), digital library was provided for use in classroom assessment (e.g., CT, DE),
and one state (Utah) provides a formative assessment online system for use by districts.
Table 5
High School Mathematics Assessments
Tables 6-8 provide lists of the state high school assessments in mathematics given in 2015-16.
The reporting on EOC assessments in Table 6 shows that eight of the 20 states have required end
of course assessments, primarily at the level of Algebra 1/ Math I/Integrated Math 1, and
Geometry/Math 2/ Integrated Math 2. Six of the states with EOC high school assessments
require passing the assessment as a graduation requirement. Seven of these states define college
and career readiness as an intended use, and only two states report that the assessment is
intended for use in student placement.
Table 7 reports the list of state high school Comprehensive assessments which cover multiple
math subject areas. Of the 20 reporting states, 16 have a policy requiring a Comprehensive HS
NORC | State Policies for Science and Mathematics Education: Year One Results
6
assessment in 2015-16. (Thus, four states provide a Math EOC assessments and a
Comprehensive HS assessment in math.) Only three states require passing their assessment for
graduation, 11 states define intended use for college or career readiness, and one state intends a
use in placement. The Table indicates the grade in which the Comprehensive assessment is
given (grade 11 primarily). Across the 16 states, seven are using the ACT, three the SAT, four
use the Smarter Balanced assessment, and two states require their own in-state assessment.
In Table 8 states list additional HS math assessments, and 10 states report at least one additional
assessment is supported for use at the local level or by students. The additional assessments
include college and career readiness options for assessment Kentucky in 12th grade (e.g., ACT
WorkKeys, ACT Compass, and KYOTE from University of KY), diagnostic assessments (PA),
interim benchmark assessments (Smarter Balanced, Utah), EOC assessments (NC, DE), and AP
and IB assessments in District of Columbia.
Insert Tables 6, 7, 8
K-2 Assessments
States were asked to report on state administration or support of assessments for students in
kindergarten, grade 1, or grade 2. Seven states reported one or more assessments that can be
used at one of these grades, although none are required for all students. State contacts may be
able to provide further information about the content of these assessments.
Insert Table 1
Elementary & Middle Science assessments
The state assessments listed in Table 11 for elementary science show that each of the 20 states
reporting for 2015-16 used a different science assessment instrument: one state (AR) used ACT
Aspire, one used Stanford 10 (KY), one state (VT) is part of the New England Common
NORC | State Policies for Science and Mathematics Education: Year One Results
7
Assessment program, and 17 states listed an assessment specific to their state. (Note: The
NORC survey did not include information on assessment contractors by state.)
State Name of Assessments
Used for School
Accountability?
Used for Teacher
Accountability?
Used for Student
Accountability?
Used to Inform
Instruction/ Curriculum?
Grades Assessment
Administered?
Arkansas ACT Aspire Yes Yes Yes Yes 3rd–8th
Connecticut Connecticut Mastery Test in Science
Missing Missing Missing Missing 5th and 8th
Delaware DCAS Science
Yes No No Yes 5th and 8th
District of Columbia
DC Science Assessment
No No No Yes 5th and 8th
Idaho
Idaho Standards Achievement Test (ISAT) for Science
Yes Yes No Yes 5th and 7th
Also shown in the table, 16 states assess science currently at two grade levels, generally grade 4
and 7 or grade 5 and 8, while two states assessed science at each of the grades 3 through 8, one
assessed grades 3 through 7, and one grades 4 through 8.
Insert Table 11-12
Of the 20 states reporting elementary/middle (grades 3-8) science assessments, 14 states reported
the intended use of results for school accountability, 10 states reported intended use for teacher
accountability, three states intend to use science results for student accountability, and 14 states
intended the assessment results be used to inform curriculum and instruction in science.
Responses from state education specialists regarding examples of how data are used locally
focus mainly on informing decisions about curriculum and instruction and aligning instruction to
standards.
In four states, support was provided for additional science assessment options that are available
for local application, as reported in Table 12. Utah provides formative and diagnostic science
assessments, Pennsylvania has diagnostic assessments, and North Carolina provides final exams
for elementary science.
NORC | State Policies for Science and Mathematics Education: Year One Results
8
High School Science Assessments
Tables 13 - 16 provide lists of the state science high school assessments for 2015-16. The
assessment reporting is organized by end of course (EOC) assessments and Comprehensive
assessments. In Table 13 the reporting on EOC science HS assessments shows that 13 of the 20
states have end of course assessments, primarily at the level of Biology 1 and several states have
assessments for chemistry, physics, and technology/engineering, and earth science. Eight states
require passing at least one EOC high school assessment for course credit for graduation; and
several states require passing a combination of at least two science EOCs. Of the 13 states, 11
have school accountability as an intended use, eight use them in teacher accountability, and six
states define college and career readiness as intended uses. Examples of use of the assessment
data in curriculum and instruction are reported in Table 14 as well as state policies on taking and
re-taking the assessments for course credit.
Table 15 lists the state high school science Comprehensive assessments which cover multiple
science subject areas. Of the 20 reporting states, 13 have a policy requiring a Comprehensive HS
assessment in 2015-16. (Thus, six states provide a Science EOC assessment and a
Comprehensive HS assessment in science.) Only one state requires passing the assessment for
graduation, while 10 states define the intended use for college or career readiness and eight of
these states use ACT for science assessment. Six states report informing curriculum and
instruction as an intended use of the results, and these are states with a state-specific science
assessment or the NE Common Assessment. The information in Table 16 reports examples of
use with curriculum and instruction and state policies on taking or re-taking the comprehensive
science assessments.
What Types of Assessment Items and Tasks?
States were asked to report on the types of assessment items or tasks that comprise the
assessment in mathematics and science for one elementary grade and for one assessment in high
school (either comprehensive or EOC). The objective in reporting on types of items is to provide
an indication of the state policy toward use of multiple methods of assessment particularly tasks
with complexity that will increase the degree of alignment of the assessment to state content
standards for mathematics and science.
NORC | State Policies for Science and Mathematics Education: Year One Results
9
The participating states reported on their use of mathematics items/tasks developed by one of the
State Assessment Consortia –Smarter Balanced or PARCC. With the sign-off of most states
with the Common Core State Standards, and the initial involvement of many states with the
planning and development of mathematics assessments with one of the two state consortia,
interest is high in tracking the current assessment providers for states. Interest is also high
because states have changed their policies over the past years and decided to contract with other
developers of assessments in math. The technical advisers for the NORC project recommended
questions to track the degree of use of items/tasks developed through the state consortia. To
reduce burden, the reporting for 2015-16 focused on item/task design for two grade levels
(elementary—grade 5 math, grade 4 or 5 science, and high school – the most advanced
assessment completed by the most students). NORC and the technical advisers decided that
information on assessments at two levels could provide important information on policy
differences and trends among the states.
Table 19 reports on the states use of mathematics assessments from the Smarter Balanced or
PARCC consortia for 2015-16 assessments. Half of the 20 participating states used assessments
at the elementary or high school level. Other states provided comments on the development of
math assessment items/tasks and vendors contracted.
Insert T 19
NORC | State Policies for Science and Mathematics Education: Year One Results
10
Table 19. Development of State Mathematics Assessments and Use of State Consortia Items/Tasks
State
Use SBAC or PARCC
Percentage of Items
Use SBAC or PARCC
Percentage of Items
Comments Grades 3–8 3–8
Assessment High
School HS
Assessment
Arkansas No No
Connecticut SBAC 100% No The state uses the SAT as the High School Mathematics assessment.
District of Columbia
PARCC 100% PARCC 100%
Delaware SBAC 100% SBAC
Idaho SBAC 100% SBAC 100%
Kentucky No No
Louisiana PARCC 47–49% varied by grade level
No The percentage of PARCC items used in 2015-16 forms was legislatively mandated to be no more than 49.9%.
Massachusetts PARCC 75% students took PARCC
No Districts selected PARCC or MCAS assessments for grades 3-8
Minnesota No No
Michigan SBAC 90% No High School is SAT Only.
North Carolina No No
Oklahoma No No
Pennsylvania No No
Tennessee No No Items were developed to TN state specific standards, and were licensed from SAGE, Kansas performance tasks, and a few items from the AIR bank.
Utah No No SAGE assessment was developed under a contract with AIR, including a pilot with teachers and administrators.
Vermont SBAC 100% SBAC 100%
Washington SBAC 100% SBAC 100%
Wisconsin No No The Forward math items for grades 3–8 designed by Data Recognition Corp. ACT, Aspire, and Work Keys used for high school.
West Virginia SBAC 100% SBAC 100%
Notes: For the state administered grades 3–8 Math assessment, does the assessment use SBAC or PARCC items? and, if so, Indicate percentage that are SBAC items and PARCC items. For high school Math, same questions.
NORC | State Policies for Science and Mathematics Education: Year One Results
11
Elementary Mathematics Items/tasks
The data in Table 20 report the results on percentage of assessment points for grade 5
mathematics by type of item or task. The technical advisers and feedback from states from the
pilot study produced a list of 11 categories of assessment item or task types. One change from
the pilot study was to provide more specific categories describing the task or item, rather than
general categories such as performance task, technology enhanced, or select response. The
reporting categories for types of math items or tasks were the following:
Multiple choice, single correct response Multiple choice, multiple correct responses
Matching tables
Drag and drop
Table Fill In
Gridded response
Graphing
Equation/Numeric
Hot Spot
Short answer
Extended response
Across the states, three of the 20 reporting states (KY, MS, OK) reported using only multiple
choice, single answer items. Two states used two types of assessment items, and nine states used
four or more types of assessment items or tasks. Five states reported that the assessment
item/task types could not be tabulated from the state assessment blueprint or framework, or the
assessment used a computer-adaptive platform (Utah). The states using Smarter Balanced
items/tasks at grade 5 (seven states) and Minnesota used computer-adaptive assessment
platforms but were able to provide estimates of the range of assessment points across students for
each category of item type or task. The reported policies on types of grade 5 math items and
tasks indicate that the states participating in the two assessment consortia are assessing students
with multiple methods and wide range of item and task designs, and thus these assessments
address a broader range of mathematics knowledge and skills in applying knowledge than other
states’ assessments.
NORC | State Policies for Science and Mathematics Education: Year One Results
12
Insert table 20
High School Mathematics Items/tasks
The state reporting on types of items/tasks shown in Table 21 indicates a divergence among the
states in how high school assessments are constructed. Six states reported that all high school
assessment items were multiple-choice, single response items, and five of these states had
adopted the ACT as the required high school assessment. Assessments in seven states include
four or more types of items/tasks, and six of these seven states were members of the assessment
consortia. Two states used two types of assessment items (using SAT as the high school
assessment). At the high school level, four states reported that the assessment item/task types
could not be tabulated from the state assessment blueprint or framework, or the assessment used
a computer-adaptive platform (Utah). Similar to the grade 5 assessments, the states using
Smarter Balanced high school items/tasks and Minnesota used computer-adaptive assessment
platforms, and they were able to provide estimates of the range of assessment points across
students for each category of item type or task. The reported policies on high school
assessments’ types of items/tasks indicate that the states participating in the two assessment
consortia are assessing students with multiple methods and a range of item and task designs, and
thus these assessments address a broader range of mathematics knowledge and applications than
other state assessments.
Insert T 21
Table 22 lists the comments about the state Mathematics assessments provided by state
specialists. The comments include explanation of the use of estimated ranges by category for the
computer-based adaptive assessments, and the development of items for the assessments.
Elementary Science Items/tasks
As with mathematics, the types of reporting categories for types of science items and tasks were
expanded following the results from the pilot study. A set of 12 categories were recommended
for science assessment. For example, the performance tasks category was expanded to three
categories—interactive computer tasks, simulations, and hands-on tasks. describing the task or
NORC | State Policies for Science and Mathematics Education: Year One Results
13
item, rather than general categories such as performance task, technology enhanced, or select
response. The science categories are the following:
Multiple choice, single correct response Multiple choice, multiple correct responses
Matching tables
Drag and drop Fill In blank Hot spot Short answer Extended response Interactive computer tasks Simulations Hands-on tasks
Gridded response
The types of science assessment items or tasks used across the 20 states (Table 23) were
predominantly multiple-choice, single-answer items in 2015-16. In eight states, all (100
percent) of the points on the state assessment for grade 4 or 5 were based on multiple-choice,
single answer items, and two states had 95 percent of points from this item type. Four states
assessments incorporated three or more types of science items/tasks in the elementary assessment
(DC, MN, WA), and three states included extended response science items (CT, DC, MA) that
comprised a significant portion of the assessment. Five states did not report on the types of
science items/tasks.
Insert T 23
NORC | State Policies for Science and Mathematics Education: Year One Results
14
High School Science Items/tasks
The state Science assessment items/tasks for 2015-16 were primarily multiple choice, single-
answer items, with eight states assessment comprised of only this type of item. Nine states
reported on Biology 1 EOC assessment as the highest level science course and assessment
completed by most students. Only three states used more than three types of items or tasks, and
three states had extended response items. Two states (DC, MN) were using technology enhanced
items through the computer. Minnesota used hot-spot and simulation tasks in the high school
assessment. One state (MI) reported that the state pilot tested-based technology enhanced items
in 2015-16. Six states did not report on the types of high school science items/tasks.
Insert T 23 a
Assessment Administration and Reporting Policies
Computer Technology in Administering Assessment
Increasingly, states are moving towards computer technology to administer student assessments.
[In mathematics, five states use a computer-adaptive format. In science, only one state uses
computer-adaptive format. Only two of the pilot states administer the math assessments solely in
a paper-and-pencil format, and one state administers science in paper and pencil format. Three
states administer assessments with computer technology and two of these states also offer the
paper-and-pencil format.
[results in preparation: One question not resolved by the pilot study was the extent to which
methods of administration are actually used, since in some states both paper-pencil and
computer-based platforms are in place. It would be useful to track the percentage of students
assessed with each method.]
NORC | State Policies for Science and Mathematics Education: Year One Results
15
Dates for Administration
In 2015-16 a majority of the 20 participating states administered assessments close to the end of
the school year (e.g., late May). One state in the pilot study conducting their assessments in
March. Policies on timing are important in relation to the total time for teaching and learning
prior to the assessments. The information should be reported and tracked over time by state.
Guidance on time for completing assessment
NORC collected information on state policies regarding time allowed to students for math and
science assessments. Most of the 20 states provided guidance about expected time for
completion, however only a few states had state requirements limiting time for assessments. A
revised version should address the question of what state guidance is provided to districts and
schools with respect to the expected time needed for most students to complete the assessment.
Use of Calculators in Assessment
State policies vary widely on the extent to which calculators are allowed for use in mathematics
assessments. The results of reporting on calculator use in math assessment for 2015-16 in Table
25 show that some states do not allow calculators use in the elementary assessments (e.g.,
Smarter Balanced states), while other states allow calculators on a portion of the assessments,
and six states have no policy at the elementary level. At the middle level, a majority of states
allow calculators use on 40 to 60 percent of the items. In high school math assessments, eight
states allow calculators on the entire math assessment, and nine states allow the calculator on
about half of the state assessment.
In the 2015-16 reporting, calculator use in science assessment was added to the policies survey.
The reporting in Table 24 shows that many states do not have a policy on use of calculators in
science assessments (shown as NR). A majority of states allow use of calculators in science at
each grade level, and in the states with computer-based assessments (e.g., ID, UT) the tool is
provided when it is suggested for use on specific items.
Insert T 25, T 24
NORC | State Policies for Science and Mathematics Education: Year One Results
16
Release of Assessment Items/Tasks
One method of providing educators with additional information and professional development
regarding assessments is to release assessment items or tasks. In 2015-16, 8 states released a
portion of science items/tasks from a prior year. The results (Table 27) show that states that
release items generally provide a small percentage of the actual test items, and some states share
similar example items. Massachusetts stands out by releasing 50 percent of the grades 3-8 items,
and 100 percent of high school science and engineering/technology EOC assessment items. Nine
states reported a website that is provided for teachers and educators to access example student
assessment items.
Insert Table 27
Mathematics items were released in only six of 20 states and a portion of the assessment
items/tasks were made available to the public (Table 26). Nine states provided example items on
a website designed for teachers and educators.
Timing of Reporting on Assessment Results
The participating states reported on the time period when math and science assessment results
are reported for three different purposes: 1) use of data by teachers, 2) review by students and
parents, and 3) school accountability. The month when the assessment results are reported for
these purposes are shown in Table 28. The amount of time between administration of tests to
students and the scoring, analysis, and reporting of results is one of the primary reasons for state
policies to favor computer-based assessments and use of computers in scoring and tabulation of
results. The use of assessment scores in school accountability has become another major state
policy issue and particularly in providing the status of schools meeting performance goals prior
to the start of the next school year. The results show that many states are providing quick
turnaround of scores for students and parents following spring testing, and the school
accountability reporting is provided in early fall for most states.
Insert T 28
NORC | State Policies for Science and Mathematics Education: Year One Results
17
Content of Mathematics and Science Assessments: What is the degree of
alignment to Standards?
The Math-Science Reporting system is being supported by NSF in relation to the recent support
for improved STEM K-12 education indicators. The series of EAGER grants from NSF were
intended to support design and research for these indicators, and NORC completed a design
study for a state assessment policy indicator and it was awarded in light of the strong interest in
how states are responding with assessments that would be consistent with the Common Core
State Standards in mathematics and science assessments that respond to the Next Generation
Science Standards.
For the implementation of an assessment reporting system for 2015-16, two strategic decisions
were made by NORC in the instrument design with the Technical Advisers panel. First, we
decided it is important to identify the official policy position of each state concerning the CCSS-
M, and the relationship of the state standards to CCSS-M. Then, states would be asked several
questions regarding assessment design and development that would provide indicators of the
degree of alignment to mathematics assessments to a common set of mathematics content topics
at the elementary and high school levels. Second, for science, we decided to similarly track the
official state policy position concerning the Next Generation Science Standards (NGSS) and the
NRC Framework for K-12 Science Education –that is, whether state science standards have been
developed or revised in relation to these documents. Then, a series of questions would be given
to SEA contacts to determine whether science assessments are being developed in response to
the key themes of these documents, particularly since 2013 when the NGSS was released by the
consortium of states developing the standards.
NORC | State Policies for Science and Mathematics Education: Year One Results
18
Mathematics Assessment Content
The question of state policy on mathematics standards and the Common Core Standards is
addressed with the results from 20 participating states in Table 19. Eleven of the states reported
that their state standards are the same as the CCSS-M, and seven states made minor revisions in
CCSS-M for their state math standards. Two states (MN, OK) developed standards separately
from the CCSS-M.
Insert T 29
Mathematics Assessment Content Topics, Grade 5
To analyze math content in assessments across states, states were asked to report on the
percentage of the grade 5 assessment that focuses on each of six topics, and the selected topics
are reporting are consistent with content recommended in the Common Core Standards. These
topics provide a way to provide comparable data across states. Additionally, states were asked to
report on one additional topic for grade 5 assessment emphasized in their state. Grade 5 was
recommended by the expert panel because it is a key transition grade between math instruction in
the elementary grades K–4 and middle grades mathematics instruction. Example data are below.
State Name of Assessment
Numerical Expression
Add/sub Fractions
Multi/div Fractions
Multi-digit
Volume Add/ mult.
Graph 1st
Quad Data
Distrib. Add.
Topic**
% points % points % points %
points %
points %
points %
points %
points
Connecticut CT Smarter Balanced Assessments
0-15 5 to 20 5 to 25 5 to 20 5 to 20 5 to 20 0 5 to 20
District of Columbia
PARCC 18-23 11 to 38 17-44 12 to 36 8 to 29 5 2
Delaware Smarter Balanced
0-15 5 to 20 5 to 25 5 to 20 5 to 20 5 to 20 0 5 to 20
Idaho ID Standards Achievement by SBAC
0-15 5 to 20 10 to 25 5 to 20 5 to 20 5 to 20 0 5 to 20
Kentucky K-PREP (Vendor-Pearson)
15 5 2 13 12 5 0 30
Louisiana LEAP 3 8 6 8 6 3 2 8
Minnesota
MN Comprehensive Assessments, Series III
2 to 15 6 to 15 0 24-50 2 to 14 2 to 9 14-17 7 to 10
NORC | State Policies for Science and Mathematics Education: Year One Results
19
Insert Table 30 – 31 for all 20 states
The percentage of grade 5 assessment points reported show that the six topics are being assessed
across the 20 states, and that percentage of the assessment varies widely for each topic. For
example, the topic “Write and interpret numerical expressions” varies from 0-15 percent to 26
percent of the grade 5 points. The percentages are a range for about half the states. Due to the
adaptive test design in many of the reporting states, the percentages are reported as a range since
students see different combinations of assessment items and topics based on their performance.
Across the six topics, a total of 50 to 80 percent of the grade 5 points are included. The
additional topic included for grade 5 included early geometry concepts, measurement, and real-
world applications.
Levels of Cognitive Demand, Grade 5 Math
Analysis of content alignment of standards and state assessments has typically included both
content topics and cognitive demand expectations (or levels of complexity). Many states refer to
the “depth of knowledge (DOK)” levels. Based on the design pilot study, the states were asked to
report on what cognitive levels are used in developing the assessments and whether the levels are
used in reporting assessment results. The information was collected for the grade 5 assessments.
Insert T 32
The policies reported by 20 states show that all of the states use some levels for the development
of math assessments. Many states plan their assessments with three cognitive levels. Less than
half indicate that the cognitive levels are used in reporting assessment results. Comments provide
further information about how levels are used and weblinks to the state definitions.
Elementary Mathematics Progression of Content Topics
Consistent with providing indicators of mathematics assessment content, states reported on the
grades when topics that are specified in the CCSS-M are first assessed. This indicator addresses
the issue of content progression as taught through the elementary grades, and the indicator can
compare the progressions in assessments used across the states. Results from seven of the states
across the elementary topics are shown below (see Table 33).
NORC | State Policies for Science and Mathematics Education: Year One Results
20
State/Topics* Fractions Word
Problem Multiply
Fractions Volume Divide
Fractions Proportions
Connecticut 3 3 5 5 6 7
Delaware 3 3 5 5 6 7
District of Columbia
3 3 4 5 6 7
Idaho 3 3 4 5 6 7
Kentucky 3 3 4 5 6 7
Louisiana 3 3 5 5 6 6
Mass. 3 3 5 5 6 6
Across all of the 20 states (Table 33), the responses show high consistency on assessing three of
the topics—understanding fractions, word problems with multiplication/division, and dividing
fractions. The grades differ slightly for multiplying fractions (grade 4 to grade 6) and using
proportions (mostly grade 7). Overall, the reporting on elementary topics assessed between
grades 3 and 7 show a high degree of consistency across the states.
Content of High School Mathematics Assessments
The Technical Advisory panel recommended that NORC request assessment content information
for the mathematics assessments that most students take in the most advanced assessment taken
by most students (often about grade 11). The reporting involved two steps: first, what
percentage of high school students take each of the assessments offered—either a
Comprehensive assessment or an EOC assessment, and the state policy on a minimum score
required was also reported. Second, for the Comprehensive Assessments, states were asked to
report on the percentage of the assessment points that are addressed by each of seven topics. One
additional topic was requested and the percentage covering that topic. The topics for reporting
were selected from the recommended mathematics content in the high school mathematics
Common Core Standards.
The results from the first question on what assessments students take in high school in Table 34
show that a majority of the 20 states now require a Comprehensive Assessment in math, and
more than 95 percent of students take the assessment in the typical state. Eight of the reporting
states require students to complete an EOC assessment, with five requiring Algebra 1
assessment, one requiring Algebra 2, and two requiring Geometry or Secondary math II
assessment. Only six states require a minimum score on the math assessment be attained for
NORC | State Policies for Science and Mathematics Education: Year One Results
21
students to be able to graduate from high school. These assessments have replaced the basic
skills or minimum competency tests required formerly by many states.
Eleven states reported on the content topics by percentage of assessment points in the
Comprehensive High School Math Assessment used by their state in 2015-16. The content
topics analyzed are:
a) Applying trigonometry to solve problems
b) Summarizing & interpreting data and making inferences
c) Transformational geometry
d) Setting up and solving systems of equations and inequalities
e) Building, interpreting, and analyzing functions in different representations
f) Interpreting and working with structure in equations and expressions
g) Working with complex numbers.
The percentages reported by 11 states (Table 35) show that the Smarter Balanced
Comprehensive assessment and the ACT were used by a majority of the states, with two states
reporting on the SAT and one state (MN) reporting on a specific state assessment. The
percentage of the assessment points differ significantly between these ACT and Smarter
Balanced comprehensive high school assessments on three topics—Summarizing and
interpreting data, Transformational geometry, and Working with complex numbers. The
additional high school math topics assessed by the ACT (Table 36) are Geometry and Pre-
algebra. The additional topic reported for the Smarter Balanced assessment is “Understand the
concept of a function and use function notation,” and from 5 to 20 points of the total assessment
are covered by this topic.
On reviewing the reporting of assessment content topics, the Technical Advisors recommended
that in the next version of the policies reporting instrument, further information be collected on
the assessments being reported. The end of course assessments in mathematics are not given for
a specific grade, but rather at completion of a high school course. They recommended that states
report on content of the most advanced high school math assessment that most students take
before graduating, either a comprehensive assessment (e.g., the HS assessment offered by the
NORC | State Policies for Science and Mathematics Education: Year One Results
22
Smarter Balanced consortium) or the highest level end-of-course assessment most students take
in high school.
Math—Formative Assessments
Science Assessment Content
The reporting on science assessment content first examines the state policy position concerning
the NGSS and the NRC Framework, and if standards are being revised or updated in relation to
these documents. From input of the advisory panel, reporting on specific content alignment
indicators will wait until more states have made a transition. For 2015-16 reporting, states were
asked to report on the timing and nature of the transition to new assessments.
The results reported on state policy position and standards (T38) indicate that 11 of the 20 states
adopted new science standards since 2013. Among the 11 states:
Seven states adopted the NGSS standards as a whole, without changes
Three states have standards that are “similar” to NGSS
One state made revisions to NGSS before adopting
Additionally, seven states have a plan and timeline for adopting new state science standards
between 2016 and 2019. The table provides comments from states reporting on their adoption
process as well as weblinks to the state standards.
State Planning for New State Science Assessments
In responses to questions on state plans for New Science Assessments, a series of questions
addressed the timing, content, and organization of development of new assessments. First,
(Table 39) states reported on the whether there is a plan for new science assessments, the target
date, states in collaboration, and comments about the process. The modal year for implementing
new science assessments is 2018-19 for these 20 states. Only a few states reported on
collaborations with other states.
Next, (Table 40) states reported on the question of planning for assessing three dimensions of
science content and strategies for planning content of new assessments. Several states reported
they are planning assessment clusters or bundles in order to assess across the three dimensions.
Fourteen states reported that their plan calls for assessing students on the three dimensions of
NORC | State Policies for Science and Mathematics Education: Year One Results
23
content, with a majority indicating that the dimensions will be assessed together. Following are
examples of state reporting on their plans for new assessment design and use of sampling to
assess the three dimensions.
State
State Approach to Design and Use of Sampling
Connecticut Some use of assessment item clusters. Sampling uncertain at this time.
District of Columbia
Outlined in blueprint
Delaware Our approach is modeled very closely with the recommendations of the BOTA report. Beyond that we are in developmental stages of our plan and not yet ready to share further information.
Idaho Currently still in the planning stage
Kentucky Bundled PEs around a phenomenon will be used to develop a cluster, in which each item will assess as least 2 dimensions.
Louisiana Unable to answer until we have revised standards
Minnesota Currently researching future approach. Test specifications after new standards adopted.
Michigan
Working with the topic bundles as the PE bundles for assessment and using the CCCs, DCIs, and SEPs, as elements that can be used in any combination to address the phenomenon. Matrix sampling will allow multiple Topic bundles to be assessed over the state.
States also reported on the planned organization of new assessments (Table 41) including the
grades to be assessed, use of grade span or grade specific assessments, and high school science
assessment plans. About half the states were still making decisions about the organization of the
grades and structure of the new assessments as of reporting in June 2016.
Current Reporting of Science Assessment Results
An additional indicator of content of science assessment is how results are reported. States were
asked to report on sub-scores and achievement levels for 2015-16 science assessments. The
results (T42) show that 11 states gave categories for the science sub-scores with varying number
of sub-scores. Additionally, 13 states reported that achievement levels are defined and reported
for the science assessments, and the websites for the levels are provided. Examples of science
reporting categories are shown below.
NORC | State Policies for Science and Mathematics Education: Year One Results
24
State Report
Sub-scores Science Categories
Report Achiev. Levels
Arkansas No
Connecticut Yes
Life Science; Physical Science; Earth Science; Content Knowledge; Scientific Inquiry, Literacy and Numeracy
Yes
District of Columbia
New Based on domains Not yet
Idaho Yes Nature of science, life science, physical science, earth science, and technology
Yes
Kentucky No
Louisiana Yes Life Science, Physical Science, Earth and Space Science, Science in the Environment, Science as Inquiry
Yes
Massachusetts Yes Earth and Space, Life, Physical, and Technology/Engineering
Yes
Minnesota Yes Nature of Science and Engineering, Physical Science, Earth Science, Life Science
Yes
Michigan Yes Domain level (Life Science, Physical Science, Earth Science, Inquiry)
Yes
Formative Science Assessments
State leadership with use of classroom-focused formative assessments is an additional measure
of state assessment activity. Nine of the 20 reporting states (Table 43) described a current
initiative or plan for supporting use of formative assessments or professional development with
teachers on formative assessment.
NORC | State Policies for Science and Mathematics Education: Year One Results
25
Rationale for a STEM K-12 Indicator on State Assessment Policies
In 2015, NORC at the University of Chicago (NORC) submitted a proposal to the National
Science Foundation program on Promoting Research and Innovation in Methodologies for
Evaluation (PRIME) of the Division of Research on Learning. A three-year grant was awarded
to NORC for the project, STEM Indicators: Implementing a Model for Reporting and
Research on State Assessment Policies for K-12 Science and Mathematics Education
(#1544123).
The goal of the project is to develop and implement an online system for reporting state
assessment policies across states, and a new grant has been funded to extend the pilot work into a
three-year implementation project. The reporting system will inform mathematics and science
educators and leaders at state, local, and national levels about the status of state assessment
policies across the nation and provide comparable policy-relevant information that can be
tracked as changes are implemented over time. NORC is implementing the system of reporting
through an online, web-based instrument that provides accessible information across the states
and regular updates.
The STEM Indicator on state assessment policies reflects the strong interest in advancing the
quality of student assessments in mathematics and science education. Research and analysis of
the content of state achievement tests has yielded concerns about their academic rigor, as the
items tend to assess lower-level cognitive skills (such as recall, recognition, and application of
procedures) as opposed to higher-level cognitive skills (such as analysis, evaluation, and
synthesis of ideas) (Darling-Hammond, et al., 2013; Resnick, et al., 2003; Webb, 2002). Even in
the NCLB era, with its emphasis on accountability, researchers have found state tests generally
assess lower-level cognitive skills. In a recent study of tests in 17 states that were selected
because the tests were reputed to be more cognitively demanding, Yuan and Le (2012) found that
fewer than 2 percent of mathematics items assessed a higher level of cognitive demand. Using a
different methodology on 19 state tests, Polikoff, Porter, and Smithson (2011) found that only 7
percent of mathematics items required students to use higher-order cognitive skills, and fully 80
percent of mathematics items assessed memorization, recall, and use of routine procedures.
NORC | State Policies for Science and Mathematics Education: Year One Results
26
The lack of test items assessing more advanced analysis and knowledge has resulted in reforms
to state content standards and assessments, with the most prominent reform being the
development of the Common Core State Standards (CCSS-M) for mathematics and the Next
Generation Science Standards (NGSS) for science (NGA/CCSSO, 2010; Achieve & Lead States,
2013). The CCSS-M and NGSS have the potential to improve the degree to which deeper
learning is assessed through state achievement tests. The CCSS-M establishes a single set of
educational standards for kindergarten through 12th grade that identifies the concepts and
knowledge that students should acquire to show that they have attained the skills necessary for
college and career success (Yuan & Le, 2014). Similarly, the NGSS initiative focuses on
coherency and progression of core conceptual science skills from early grades through high
school, with an emphasis on preparing students for college, career, and citizenship. A recent
study report and recommendations from the National Research Council is informing the
development of new designs and systems for science education in the states to support the goals
of the NGSS (NRC, 2013).
A key issue analyzed in the design study was how to track states use of innovative features of the
mathematics assessments being provided through the consortia--Smarter Balanced (SB) and
PARCC, including performance tasks (with multiple steps and explanation of work), balance of
items across the different levels of depth of knowledge, use of computer-based testing to
improve turnaround time for scoring and reporting, measures of learning content and skills, use
of adaptive testing to measure a full range of student knowledge, benchmark testing to track
progress through the year, and a digital assessment library for formative classroom assessment.
To work toward an indicator of state assessments alignment to standards especially relative to
content coverage and levels of cognitive demand, NORC and the advisers considered the Criteria
for High-Quality Assessments defined by the chief state school officers (CCSSO, 2014).
Reporting of state policies and trends over time is equally important for science education as part
of the proposed state policies tracking system because of the recent shifts in science education
standards and the strong interest in new forms of student assessment to match the direction of the
standards. Some states are following the recommendations of the NRC committee on NGSS
science assessments (NRC, 2014), including moving toward a system of science assessment with
NORC | State Policies for Science and Mathematics Education: Year One Results
27
varied methods, assessing the three dimensions of science instruction, and maintaining annual
reporting of student progress.
Results of Design Pilot Study
The NORC plan for a design and pilot study was intended to test a model for collecting
information on states’ policies for state-administered student assessments to meet several needs.
The pilot study (2014–15 school year) tested a design for collection of state policies information
from three sources—state websites, survey of state assessment staff (focusing on design and
methods of reporting), and survey of state content specialists in math and science (focusing on
content and alignment of assessments and standards). The design options and issues of focus in
the pilot study are summarized in a paper, developed from the recommendations of a project
expert panel (NORC, 2015).
The design study and pilot project addressed two questions about state assessment policies for
mathematics and science and development of a system for tracking and reporting across the
states:
1) What core information on state assessment programs and policies should be collected and reported across states, i.e., what information on types of assessments and characteristics of assessment
programs are important to have available in a 50-state report or online resource?
2) What is an effective method for reporting comparable information on the extent to which student assessments in math and science are aligned to state-adopted content standards, and what is the
relationship to the Common Core State Standards-Mathematics and Next Generation Science
Standards?
The design pilot study was carried out with support and participation by staff of ten state
departments of education. The results of the design study were reported by NORC to NSF and
the report is available (http://www.norc.org/Research/Projects/Pages/developing-a-system-for-
tracking-state-assessment-policies-in-science-and-mathematics-education.aspx).
http://www.norc.org/Research/Projects/Pages/developing-a-system-for-tracking-state-assessment-policies-in-science-and-mathematics-education.aspxhttp://www.norc.org/Research/Projects/Pages/developing-a-system-for-tracking-state-assessment-policies-in-science-and-mathematics-education.aspx
NORC | State Policies for Science and Mathematics Education: Year One Results
28
Selection of State Assessment Policies for the Reporting System
The project Technical Advisers (see Exhibit 1) reviewed the findings of the pilot study report
and the research on needs for information on assessment policies in math and science education
to recommend implementation of a reporting system beginning with the 2015-16 school year. A
key decision in planning for a state assessment policies indicator is the degree to which reporting
and analysis across states should be comparable and quantifiable. For purposes of cross-state
reporting, a common metric and organization of information in tabular format facilitates
comparisons; however, with greater depth of information, key differences in state policies and
design of state assessments could be highlighted and shared among users in different states. It
was decided that some policies related to assessments should not be included in this reporting
system. For example, school accountability has been the primary focus of uses of state
assessments recently, however, the advisers recommended that the reporting system should
include the different types of intended uses of state assessments in mathematics and science, and
that this reporting system should forego details about state accountability plans and the formulas
by which schools, teachers, or students are determined to meet expected performance levels.
A) Characteristics of State Assessments in Science and Mathematics
The panel recommended collecting and reporting information for the following categories of
state assessment policies:
■ List of types of state assessments by grade level
■ Intended uses of assessment data
■ Item or task design
■ Timing and methods of administration
■ Methods of assessment reporting and dissemination.
B) Relationship of State Assessments to Standards
The pilot study examined several options for reporting measures of the degree of alignment of
mathematics and science assessments in relation to content standards. The central question in
studying alignment is the degree to which assessments are consistent with the content and depth
of teaching and learning called for in the standards (Webb, 2002; Polikoff, Porter, and Smithson,
NORC | State Policies for Science and Mathematics Education: Year One Results
29
2011). Many states have conducted alignment studies for the assessment instruments used for
accountability reporting as part of the requirements under federal No Child Left Behind funding,
and a recent study has reported new information on the quality of the state consortia assessments
based on the Common Core State Standards and two other assessments used by states, based on
the criteria defined by CCSSO (Doorey & Polikoff, 2016).
For this project, NORC determined that it would not be possible to conduct an analysis of the
alignment of each state’s assessments in math and science. Instead, the advisers recommended a
design for collecting information on several measures of the degree of alignment that could be
compiled and reported across states that would address key differences and similarities in the
state assessments’ content breadth and depth relative to standards. A design for reporting on
grade-specific content topics, cognitive levels of assessment items, and the range of methods
used in assessing content knowledge and practices was tested in the pilot study, and with some
changes and additions, the measures were implemented for the year-one reporting for 2015-16.
The measures used are designed to provide analysis and comparison of state assessments’
content rigor and breadth specific to a grade, and the degree to which the items/tasks address the
depth of knowledge and cognitive demand of the standards. Thus, regardless of the standards
defined by state policy, and the assessments that are selected by states, the alignment indicators
should provide a common basis for reporting across states.
The following indicators of content alignment were identified for testing in the pilot study:
■ State policy on content standards, and relationship of state standards to CCSS and NGSS
■ When standards were developed and approved
■ Source of state assessment framework or blueprint (consortia, state, other)
■ Percentage of grade 5 mathematics assessments that cover selected content topics
■ Levels of cognitive demand used in developing math assessment items/tasks, and use of
levels in reporting student results
■ Elementary grades at which selected topics were assessed
■ Percentage of high school math assessments that cover selected content topics
■ State plans and methods for development of new science assessments
NORC | State Policies for Science and Mathematics Education: Year One Results
30
■ Major content topics for new science assessments.
■ Content for new science assessments aligned to standards
NORC | State Policies for Science and Mathematics Education: Year One Results
31
Summary: Year-One Results and Recommendations
Process Recommendations
Changes and edits to Questions
Cross-state results
NORC | State Policies for Science and Mathematics Education: Year One Results
32
References
Achieve & Lead States (2013). Next Generation Science Standards: By States, For States. Washington,
D.C.: National Academies Press. http://www.nextgenscience.org/next-generation-science-
standards
CCSSO’s (2014) Criteria for Procuring and Evaluating High-Quality Assessments
http://www.ccsso.org/Documents/2014/CCSSO%20Criteria%20for%20High%20Quality%20Ass
essments%2003242014.pdf
CCSSO/ATP (2013) Operational Best Practices for Statewide Large-Scale Assessment Programs.
http://www.ccsso.org/Resources/Publications/Operational_Best_Practices_for_Statewide_Large-
Scale_Assessment_Programs.html
Darling-Hammond, L., Herman, J., Pellegrino, J., Abedi, J., Baker, E., Bennett, R. et al. (2013). Criteria
for high-quality assessment. Stanford, CA: Stanford Center for Opportunity Policy in Education.
Doorey, N. & Polikoff, M. (2016) Evaluating the Content and Quality of Next Generation Assessments.
Washington, DC: Fordham Institute. http://edexcellence.net/publications/evaluating-the-content-
and-quality-of-next-generation-assessments
National Governors Association Center for Best Practices and Council for Chief State School Officers.
(2010). Common Core State Standards. Washington, DC: National Governors Association Center
for Best Practices and Council of Chief State School Officers. http://www.corestandards.org/
National Research Council (2013). Monitoring Progress toward Successful K–12 STEM Education: A
Nation Advancing? Washington, DC: National Academy Press.
National Research Council. (2013). Developing Assessments for the Next Generation Science Standards.
James W. Pellegrino, Mark R. Wilson, Judith A. Koenig, and Alexandra S. Beatty, Editors;
Committee on Developing Assessments of Science Proficiency in K–12. Board on Testing and
Assessment. Division of Behavioral and Social Sciences and Education. Washington, DC: The
National Academies Press.
NORC (2015) Study Design Paper: Developing a System for Tracking State Assessment Policies in
Science and Mathematics Education Instrument Design and Pilot Study Plans, Bethesda, MD.
March.
Polikoff, Porter, and Smithson (2011) How well aligned are state assessments of student achievement
with state content standards? American Educational Research Journal, 48(4), 965–995.
http://www.nextgenscience.org/next-generation-science-standardshttp://www.nextgenscience.org/next-generation-science-standardshttp://www.ccsso.org/Documents/2014/CCSSO%20Criteria%20for%20High%20Quality%20Assessments%2003242014.pdfhttp://www.ccsso.org/Documents/2014/CCSSO%20Criteria%20for%20High%20Quality%20Assessments%2003242014.pdfhttp://www.ccsso.org/Resources/Publications/Operational_Best_Practices_for_Statewide_Large-Scale_Assessment_Programs.htmlhttp://www.ccsso.org/Resources/Publications/Operational_Best_Practices_for_Statewide_Large-Scale_Assessment_Programs.htmlhttp://edexcellence.net/publications/evaluating-the-content-and-quality-of-next-generation-assessmentshttp://edexcellence.net/publications/evaluating-the-content-and-quality-of-next-generation-assessmentshttp://www.corestandards.org/http://www.nap.edu/catalog.php?record_id=13509http://www.nap.edu/catalog.php?record_id=13509http://www.nap.edu/openbook.php?record_id=18409
NORC | State Policies for Science and Mathematics Education: Year One Results
33
Resnick, L.B., Rothman, R., Slattery, J.B., & Vranek, J.L. (2003). Benchmark and alignment of standards
and testing. Educational Assessment, 9, 1–27.
Schmidt, W. H., McKnight, C. C., Houang, R. T., Wang, H. C., Wiley, D. E., Cogan, L. S., et al. (2001).
Why schools matter: A cross-national comparison of curriculum and learning. San Francisco:
Jossey-Bass.
Webb, N. L. (2002). Alignment study of language arts, mathematics, science, and social studies of state
standards and assessments in four states. Washington, DC: Council of Chief State School
Officers.
U.S. Dept. of Education’s Peer Review Guidance (issued 9/25/15)
http://www2.ed.gov/policy/elsec/guid/assessguid15.pdf
Yuan, K. & Le, V. (2012). Estimating the percentage of students who were tested on cognitively
demanding items through the state achievement tests. Santa Monica, CA: RAND Corporation.
http://www2.ed.gov/policy/elsec/guid/assessguid15.pdf
NORC | State Policies for Science and Mathematics Education: Year One Results
34
Appendix: Technical Advisers
Catherine Bowler, Science & Engineering Assessment, Massachusetts Department of Education
Pete Goldschmidt, CSU-Northridge (formerly New Mexico State Assessment Director)
Brian Gong, Executive Director, Center for Assessment, Dover, NH
Robin Hill, Mathematics, Kentucky Department of Education
Tammy Howard, State Accountability Services Director, North Carolina
Peter McLaren, State Science Specialist, Achieve, Inc. (formerly Rhode Island)
Hillary Michaels, HumRRO, Education Research & Evaluation, Alexandria, VA
Jessica Mislevy, Education Research Scientist, SRI International
Brett Moulding, Science Education Professional Development Director, Utah
Tiffany Neill, Science Education Specialist, Oklahoma State Department of Education
Scott Norton, CCSSO Strategic Initiative Director for Standards, Assessment, & Accountability
Cinda Parton, Assessment Specialist, Washington State Office of Education
Jim Pellegrino, University of Illinois at Chicago, Learning Sciences Institute
William Schmidt, Center for the Study of Curriculum, Michigan State University
Ellen Whitesides, Mathematics Educator & Math SCASS, Washington, DC