Summer Scoring TrainingSmarter Balanced Mathematics

Deborah J. BryantSeptember 18, 2015

Purposes and Objectives

• Learn how the summative assessments are scored• Practice using standard handscoring tools and processes• Experience methods of applying standardized criteria to the evaluation of student work• Use scoring information to improve instruction in the classroom

How Do Educators Benefitfrom Scoring Training?

• Enhanced assessment literacy– Deeper understanding of Smarter Balanced

claims and targets– Better understanding of handscoring procedures– Opportunity to take a closer look at rubrics– Opportunity to see examples of student work at a range of score points

• Instructional Improvement– Ideas for improving instruction and evaluation of student work in the classroom

Standard Handscoring Materials

• Provided by Smarter Balanced – Standards for Mathematical Practice/Targets from the Mathematics Summative Assessment Blueprint

• Ensure raters have the tools to apply the scoring criteria consistently• Include the following:

– PT sets: stimuli and prompts– Scoring rubrics and exemplars– Samples: responses with scores and

annotations– Practice set: sample responses for you to score

Summative Scoring at ETS

• Training—Summative scoring training is similar to the summer scoring training.• Certification—Prospective raters must take and pass a certification test on a set of prescored responses.• Calibration—Before raters start scoring, they must take andpass a calibration set to demonstrate that they are ready toscore accurately.• Double Scoring—10% of responses are double scored toverify that raters are scoring consistently with the rubric.• Backrating—Scoring Leaders “read behind” raters and scoreresponses raters have scored to provide another check.

Summative Scoring (cont.)

• Scoring Leaders:– Provide feedback to raters for additional content support; help prevent score drift; serve to maintain scoring pace.– Mentor struggling raters and/or direct them to preselected sample sets for retraining.

Scoring Terms• Stimulus: Short scenario with useful information needed to solve

problems.• Prompts: The questions to which students are responding, also

called “items.”• Rubric: A standardized approach to score student responses. Math

rubrics are item-specific.• Exemplar: An example of a correct answer. It is important to note

that equivalent answers are always accepted unless specific notes in the rubric indicate that only a certain set of answers will be considered as correct.

• Samples: Student responses that are prescored and annotated. Annotations are often just as important and insightful as the rubric.

Mathematics Assessment

• Performance Task (PT): A task with six prompts based on a common stimulus or

scenario; some items will be scored based on responses to other prompts in the same PT.• Non-PT (CAT): A single task with a single prompt.• Other: Some machine-scored prompts need human scoring when administered on paper.

Sample Annotations

• An annotation is the rationale given for why a response receives full, partial, or no credit.• The annotations written for samples provide further insight into how student responses were scored. It is important to read the rubric and annotations carefully.

Scoring Dependent Items

Item 1 Response

Item 2 Response

Item 3 Response

Item 4 Response

Item 5 Response

Item 6 Response

• A Dependent Item is an item that depends on the response to a previous item(s) in the PT. We call these previous items Parent Items or Related Items.

• If the reasoning is correct on item 5, but there is a follow-through error from items 3 and/or 4, award full credit.

Scoring Dependent Items (cont.)How can the dependent item receive full credit if theparent item received partial or no credit?

If the student uses the answers he or she gave to the parentitem and correctly applies them to find a solution to thedependent item, he or she can earn full credit for the answeron the dependent item.

It is also possible that while answering the dependent item,the student realized his or her errors in the parent item andmay correctly answer the dependent item without relying onthe answer to the parent item.

Assessment Claims and Targets forMathematics Summative Assessment

Assessment Claims are broad evidence-based statements about what students know and can do as demonstrated by their

performance on the assessments.Assessment Claims

1. Concepts & Procedures-40%2. Problem Solving

40% 3. Communicating Reasoning-20%4. Modeling Data Analysis

Within each claim there are several assessment targets which really define the nature of the performance expectations.

SBAC Practice Test Scoring Guide Grade 4 Performance Task Sample

A TRIP TO THE ZOOAnna and her family go to the zoo. The gift store prices are shown in the table.

Gift Store PricesGift Price

Stuffed panda bear $9

Zoo magnet $4

Pack of 4 pens $6

Photo frame $8

SBAC Practice Test Scoring Guide Grade 4 Performance Task Sample Continued

SBAC Practice Test Scoring Guide Grade 4 Performance Task Sample Continued

• Read the rubric and the exemplar. – Think about the different types of responses that a student may provide to correctly answer

the prompt. – Discuss with your group or partner what information the student will need to answer the

prompt.• For this item, a full-credit response (2 points) includes

-stating correct gifts that Anna and Ray can buy based on the student’s response to item 217 and the restrictions in the stem AND

-explaining why there is enough money for the gifts based on the student’s response to item 217.For example: “Anna and Ray can buy a stuffed panda bear and a pack of 4 pens which will cost $13. There is $25 dollars left to buy gifts. $13 dollars is less than $25 dollars.”

SBAC Practice Test Scoring Guide Grade 4 Performance Task Sample Continued

• For this item, a partial-credit response (1 point) includes: -stating correct gifts that Anna and Ray can buy

based on student’s response to item 217 and the restrictions in the stem.For example: “They can buy a stuffed panda bear and a zoo magnet.” • For this item, a no-credit response (0 points) includes

none of the features of a full- or partial-credit response.For example: “They can buy 4 stuffed panda bears.”

This item is not graded on spelling or grammar.

SBAC Practice Test Scoring Guide Grade 4 Performance Task Sample Continued

-Individually read and score Practice Set 1.-Discuss the practice papers with your group, providing evidence from the rubric and samples when you assign your scores. Try to reach consensus on a single score for each student response.-They would share the score that SBAC assigned to the practice papers along with the reasons why.

A TRIP TO THE ZOO

A TRIP TO THE ZOO

Your Score

Group Consensus

Score

SBAC Score

Comments

Practice 1Practice 2Practice 3

Closing Activity

• Reflect on today’s session, and discuss with your table group:– Strategies you learned that are relevant to

classroom instruction and assessment– Ideas you will take back to the classroom– Other thoughts about scoring training and

professional developmentShare ideas with larger group!

Interested in Scoring?/Questions?