What Does Smarter Balanced Want Students to Know and Show?
Heather Dallas, The Curtis Center at UCLAShelbi Cole, Student Achievement PartnersJudy Hickman, Smarter Balanced at UCLA
California Math Council South ConferenceNovember 6, 2015
Why New Assessments?
Why are we changing what students need to know and be able to do?
Me: My daughter’s fingernails appear to be detaching at the base. Could that be related to the illness she had four weeks ago?
Doc: [Gets on his computer to examine current research.]
Do you want a pediatrician who relies on the most recent information or what he/she learned 20 years ago?
What does his future look like?
What skill set do his future employers value?
A Balanced Assessment System
Common Core State Standards
specify K-12
expectations for college and career readiness
All students leave
high school college
and career ready
Teachers and schools have
information and tools they need
to improve teaching and
learningInterim assessments
Flexible, open, used for actionable
feedback
Summative assessments
Benchmarked to college and career
readiness
Teacher resources for formative
assessment practices
to improve instruction
Building the Math Item Pool for an Adaptive Test
7
The CCSSM Requires Three Shifts in Mathematics
• Focus strongly where the standards focus
• Coherence: Think across grades and link to major topics within grades
• Rigor: In major topics, pursue conceptual understanding, procedural skill and fluency, and application with equal intensity
8
Mathematics topics
intended at each grade by
at least two-thirds of A+
countries
Mathematics topics intended at each grade by at least two-thirds of 21 U.S. states
Shift #1: Focus Strongly where the Standards FocusThe shape of math in A+ countries
1 Schmidt, Houang, & Cogan, “A Coherent Curriculum: The Case of Mathematics.” (2002). 9
10
GradeFocus Areas in Support of Rich Instruction and Expectations of Fluency and Conceptual Understanding
K–2Addition and subtraction - concepts, skills, and problem solving and place value
3–5Multiplication and division of whole numbers and fractions – concepts, skills, and problem solving
6Ratios and proportional reasoning; early expressions and equations
7Ratios and proportional reasoning; arithmetic of rational numbers
8 Linear algebra and linear functions
Shift #1: Focus Key Areas of Focus in Mathematics
11
Shift #2: CoherenceThink Across Grades
4.NF.4. Apply and extend previous understandings of multiplication to multiply a fraction by a whole number.
5.NF.4. Apply and extend previous understandings of multiplication to multiply a fraction or whole number by a fraction.
5.NF.7. Apply and extend previous understandings of division to divide unit fractions by whole numbers and whole numbers by unit fractions.
6.NS. Apply and extend previous understandings of multiplication and division to divide fractions by fractions.
6.NS.1. Interpret and compute quotients of fractions, and solve word problems involving division of fractions by fractions, e.g., by using visual fraction models and equations to represent the problem.
Grade 4
Grade 5
Grade 6
Shift #3: RigorRequired Fluencies for Grades K-6
12
Grade Standard Required Fluency
K K.OA.5 Add/subtract within 5
1 1.OA.6 Add/subtract within 10
2 2.OA.22.NBT.5
Add/subtract within 20 (know single-digit sums from memory)Add/subtract within 100
3 3.OA.73.NBT.2
Multiply/divide within 100 (know single-digit products from memory)Add/subtract within 1000
4 4.NBT.4 Add/subtract within 1,000,000
5 5.NBT.5 Multi-digit multiplication
6 6.NS.2,3 Multi-digit divisionMulti-digit decimal operations
Using Computer Adaptive Technology for Summative and Interim Assessments
• Provides accurate measurements of student growth over timeIncreased precision
• Item difficulty based on student responsesTailored for Each
Student
• Larger item banks mean that not all students receive the same questionsIncreased Security
• Fewer questions compared to fixed form testsShorter Test Length
• Turnaround time is significantly reducedFaster Results
• GMAT, GRE, COMPASS (ACT), Measures of Academic Progress (MAP)Mature Technology
How CAT Works (Binet’s Test)
Rating Item Difficulty
A
B
C
D
E
Percentage of 4th graders getting problems like these correct (Based on
spring 2014 data)
Enter a fraction equal to 1/3.
65%
60%
45% 30%
10%
35%
35%
40%
50%
70%
75%
17
Implementing at Scale: Item Specifications
Difficulty vs. Complexity
• The adaptive test will select items of appropriate difficulty level, but all students will see items across the full range of available complexity.
• Difficulty is about how hard the item is (i.e., how many students got it right); while complexity is about the kind of thinking a student needs to do to solve the problem.
18
How do we ensure that “difficult” items are difficult for the right reasons?
A set of item quality criteria is being applied to the evaluation of items in the Smarter Balanced bank, and one is about the time vs. information tradeoff from an item:
4a. Is the time spent on the item due to the required mathematics or due to the complexity of the item itself?
19
Will the CAT select items for a student aligned to standards in other grade levels?
• Once 2/3 of items required by the blueprint have been delivered in a student’s test, the pool may be expanded to include items with primary alignment to other grade levels for students performing at the highest and lowest performance levels– The goal is to increase precision of
measurement for very high and very low achieving students (minimize error in student’s score)
20
Why Update Item Specifications?
• Increase transparency for stakeholders. Enhancements include:– More items to illustrate each assessment target– Enhanced descriptions for how to adjust the
difficulty of items within a target– Clear labeling of which items allow calculators or
other tools– Reveal new item types planned for future pilot
testing– Remove any task models that were unsuccessful
during pilot testing phase
21
Did you know?
There are over 1,000 example items for mathematics across the Smarter Balanced grades 3-8 item specifications!
What are we waiting for? Let’s take a look!
22
23
Digging Into the Item Specifications
Did you know over 1000 math items can be found in the Mathematics Item Specifications?
• Mathematics Content Specifications
• CAT Item Specs, Version 3 (9/28/2015)
• Mathematics PT Item Specs All Grades (PDF) (5/27/14)
• Scoring Guide for Selected Short-Text Mathematics Items
http://www.smarterbalanced.org/smarter-balanced-assessments
Eliciting Problem Solving: Claim 2
Problem solving sits at the core of doing mathematics.
Proficiency at problem solving requires students to choose to use concepts and procedures from across the content domains and check their work using alternative methods.
25
Distinctives of Claim 2 Items:
• Multiple approaches are feasible or a range of responses is expected (e.g., if a student can solve a word problem by identifying a key word or words and selecting operations, then it is Claim 1).
• The use of tools in Claim 2 is intended to support the problem solving process. In some cases, students may be asked to display their answer on the tool (e.g., by clicking the appropriate
point or interval on a number line or ruler).
• Assessing the reasonableness of answers to problems is a Claim 2 skill.
26
Claim 2 Exemplars
27
Claim 2 Exemplars
28
Claim 2 Exemplars
29
Claim 2 Exemplars
30
Claim 2 Exemplars
31
Claim 2 Exemplars
32
Claim 2 Exemplars
33
Eliciting Reasoning: Claim 3
This claim refers to a recurring theme in the CCSSM content and practice standards: the ability to
construct and present a clear, logical, convincing argument.
Assessment tasks that address this claim will typically present a claim or a proposed solution to a problem and will ask students to provide, for example, a justification, and explanation, or counter-example.
34
Distinctives of Claim 3 Items:
• Items and task assessing Claim 3 may involve application of more than one [content] standard. The focus is on communicating reasoning rather than demonstrating mathematical concepts or simple applications of mathematical procedures.
• Targeted content standards for Claim 3 should belong to the major work of the grade.
35
Claim 3 Exemplars
36
Claim 3 Exemplars
37
Claim 3 Exemplars
38
Claim 3 Exemplars
39
Eliciting Modeling: Claim 4
Modeling is the process of choosing and using appropriate mathematics and statistics to analyze empirical situations,
to understand them better, and to improve decision-making
40
Distinctives of Claim 4 Items:
• Claim 4 differ from those in Claim 2, because while the goal is clear, the problems themselves are not yet fully formulated (well-posed) in mathematical terms.
• Claim 4 items and tasks should sample across the content domains, with many of these involving more than one domain.
• Because of the high strategic demand that substantial non-routine tasks present, the technical demand will be lower—normally met by content first taught in earlier grades.
41
Claim 4 Exemplars
42
Claim 4 Exemplars
43
Claim 4
Claim 4 Exemplars
45
Development Processes
Item Development Considerations
• Engaging for all students• Range of items of high and low difficulty• New item types for computer response
capture and real-time scoring• Robust tagging for interoperability
Operational Blueprint
Requirements
Field Test Item Development
Plan
Content and Item
Specifications
Tagging requirements
Claim, Target,
Task ModelItem Type
Scoring method
Cognitive Complexity
Difficulty
Accessibility, Bias/Sensitivity, other Guidelines
Specifications/Archetypes
● Item Specifications● Reflect lessons from pilot● Content reviews● Accessibility, bias, sensitivity
reviews● Archetypes
● Models● Content reviews● Accessibility, bias, sensitivity
reviews● Language Complexity Rubric
Foundation of Development Approach
Item Types
• Multiple Choice• Multiple Select• Hot Spot• Equation/Numeric• Graphing• Fill In Tables• Matching Tables• Drag and Drop• Short text
Item Quality - Supporting Activities
• Item Reviews– Occur in advance of audits– Independent but managed by CTB– Items housed in DAS and ITS systems– Comments evaluated for item revision, approvals
• Item Audits– Final quality control activity before release– Managed by Smarter Balanced– Items housed in ITS, feedback collected on
Quality Criteria
Purpose
•Define characteristics of Smarter Balanced items/tasks to ensure high quality items
•Provide item authors with quality guidelines to apply during development
•Provide item reviewers with quality guidelines for item/task review
Role
•Expand traditional item development quality guidelines
•Ensure focus on important elements of evidence and alignment
•Provide gatekeeper criteria to focus authoring and review
•Guide overall quality training and processes
Language Complexity
Cook, H. G. & MacDonald, R. (2013). Tool to Evaluate Language Complexity of Test Items (WCER Working Paper No. 2013-5). Retrieved from University of Wisconsin– Madison, Wisconsin Center for Education Research website: http://www.wcer.wisc.edu/publications/workingPapers/papers.php
Questions?
Heather Dallas: [email protected] Cole: [email protected]
Judy Hickman: [email protected]
California Math Council South ConferenceNovember 6, 2015