1

• Welcome to Unit 7 of the Assessment Literacy Professional Learning

Module.

• In Unit 6 we explored some of the elements that make quality assessments.

• In this unit we will move into using the results of assessments.

2

• Are there any questions about these objectives or any additional objectives or information you’d like covered?

Note to Facilitator: Make note of requests for additional information. As you move through the unit, address any relevant issues raised by the group.

3

• Our focus will be on how teachers and students can use data in the classroom.

[refer to slide]

4

• Here is where we are in the teaching-assessing cycle and the assessment literacy attribute we’ll focus on.

5

• By a show of hands, how many of you have used a rubric?

• This section will cover the basics of quality rubrics.

• A rubric is a scoring tool that gives assessment-literate educators and

students accurate information about current performance as well as

feedback on how to improve.

• A rubric is more sophisticated than a checklist; it also articulates gradations

of quality for each criterion.

• Let’s take a few minutes to watch Dr. Popham explain the basics of high-quality rubrics.

Video URL: http://www.k-state.edu/ksde/alp/module7/

6

• By a show of hands, how many of you have used a rubric to score a student performance other than writing? For example in science or music or PE? What were they for?

• Scoring selected-response assessments is straightforward; answers are

either correct or incorrect.

• On the other hand, scoring constructed-response assessments, such as

essays and performance tasks, requires a scoring rubric.

• Recall from Unit 2 that performance tasks were introduced and you

analyzed some Smarter Balanced sample items that included rubrics.

7

Note to Facilitator: Review the definitions on this slide.

8

• By a show of hands, how many of you have developed a rubric? What was the experience like? Would you do it again? Why or why not?

Note to Facilitator: When you open the URL, scroll to the video titled, "How Are

Rubrics Developed" on the Web page that appears.

• This video highlights using rubrics in combination with exemplars, or pieces

of work that typify the score descriptors.

• Having both descriptions and examples helps scorers interpret the rubrics

the same way, increasing consistency between raters.

Video URL: http://www.k-state.edu/ksde/alp/module7/

9

• Many times rubrics will use different nomenclature for these parts, but the parts are basically the same.

Note to Facilitator: Review slide.

10

• Let’s explore the parts of a rubric using an example titled, “Chocolate Chunk

Cookies.”

• As you review the rubric, notice that: [refer to slide]

11

• Sometimes rubrics will be organized differently for example the levels of quality will go from high to low.

• How many of seen a similar rubric? Has anyone used it with students? What was your objective in using it with students?

• Based on the rubric, how would you rate a cookie that has a chocolate

chunk in every bite and is burned on the bottom?

• Write your response in your journal. Sample Response Good chocolate chunk balance (3) with poor color (1) and appearance (1). While the Chocolate Chunk Cookie rubric is not an exemplary one, it shows how the necessary parts of rubric can be organized and used. Many assessment literate educators have used a similar exercise to introduce their students to rubrics in a fun and non-threatening way.

12

Note to Facilitator: The objective of this activity is to familiarize teachers with the Smarter Balanced writing rubrics. It is designed as a gradual release of responsibility but it can be done either individually, in small groups, or as a large group or a combination. Project the task overhead for participants to view on a large screen (http://www.smarterbalanced.org/wordpress/wp-content/uploads/2012/09/performance-tasks/animal-performance.pdf), refer participants to the Web link on the slide, have them download in advance, or provide printed copies. Display the rubric using an overhead projector for large group Guided Practice and discussion of the answers to the questions on the slides that follow. • Take a few minutes to look over the rubric on pages 17–19 of the document to

answer the questions on the next slide.

• We’ll be looking at the Smarter Balanced rubrics as a gradual release of responsibility activity.

• Let’s review the Grade 4 Smarter Balanced ELA Performance Task: “Animal

Defenses,” first presented in Unit 2. • I will show you the answers to the first two questions, then we will answer

questions 3-5 as a group.

13

Sample Responses: 1. What genre of writing is this? Informative-Explanatory 2. What grade span? Grades 3–5 Facilitator: Show where the information is found in the rubric. Let’s do the rest of the questions together as Guided Practice. 3. How many scoring levels for this item? 4 plus “NS” for No Score 4. In your opinion, are there enough levels to allow a clear distinction between the different gradations of quality? Yes, there is enough room for varying levels of quality, yet they are not overwhelming. 5. What are the criteria? “Statement of Purpose/Focus and Organization” “Evidence/Elaboration”, and “Conventions”

13

• Answer questions 6-9 with your elbow partner. Be prepared to share out. Sample Responses: 1. In your opinion, do the criteria cover important features that are relevant to this genre of writing? Explain your answer. Yes, I think so. The three criteria, “Statement of Purpose/Focus and Organization” , “Evidence/Elaboration” and “Conventions” are the most important features of writing that informs or explains. An argument could be made to include other features, but it is more important to keep the rubric simpler and easy to use for teachers and students. 2. What is the descriptor under the criterion “Statement of Purpose/Focus and Organization” for a score of 4? The response is fully sustained and consistently and purposefully focused; controlling idea or main idea of a topic is clearly stated, focused, and strongly maintained; controlling idea or main idea of a topic is introduced and communicated clearly within the purpose, audience, and task. 3. Compare a descriptor for a 4 with the corresponding 1. Are they written clearly so that you could rate student work? Explain. Yes, they clearly distinguish the level of quality that is expected. I could easily and quickly move through the ratings and identify where a student’s response fits on the

14

scale. Sometimes words like “effective” or “minimal” leave room for varying levels of interpretation, but for this type of task there will always be an element of individual interpretation. 4. How do the descriptors in this rubric compare to the “Chocolate Chunk Cookie” rubric? They are longer with more detail.

14

Note to Facilitator: This is the opportunity for independent practice. Download and reproduce (or have participants do so) the performance tasks most closely matched to the participants’ grade level assignments. Elementary teachers could tackle the mathematics rubric. Alternatively, this could be assigned as homework if you will be meeting with the group again. • On the slide are links to other Smarter Balanced performance tasks for

writing in higher grades.

• Select one that is closest to your grade span and familiarize yourself with

the task and the scoring information.

• View the page numbers indicated and consider each question presented

above as you review the rubric.

• Note that this information is useful to educators of all disciplines in

evaluating student writing, not only for ELA.

15

Note to Facilitator: This is an opportunity to apply what has been learned in the module to the participants’ practice. It could be the next activity of the module if, before the workshop, you download and reproduce the rubric and ask the participants to bring in a rubric they are currently using and like or dislike. Or, it could be assigned as homework if you will be meeting with the group again.

• Let’s take a look at a quality rubric, developed by the Kansas

Department of Education, that can be used to evaluate any rubric.

URL: http://www.k-state.edu/ksde/alp/activities/Activity7-5.pdf

16

• Now we’ll move into why students should be involved in their assessment and how it can be done.

17

• Recall from Unit 5 the many benefits of involving students in the formative

assessment process.

• Once students are familiar with how educators use rubrics, they can begin

using rubrics to score their own work.

• Educators who create a climate where students are a community of learners

can also have students score each other’s work.

18

• This list of positive research outcomes from the book, Leading the Way to

Making Classroom Assessment Work (2008), is based on students being

involved in assessing their own learning. [review slide]

• As you see, there is a strong body of research showing the benefits of involving students in assessment. Turn to your elbow partner and discuss what are some of the key words and phrases that stand out?

19

• As you watch the video, think about the questions on the slide and how this might work with your students.

Video URL: http://successatthecore.com/teacher-development/featured-

video.aspx?v=40

20

• In Unit 5, you learned that students are excellent learning resources for

each other and that peer assessment complements self-assessment.

• In this video, high school students in a math class used a rubric to rate other

students’ presentations.

• Consider how questions and answers about the logistics of peer

assessment in the video compare to how peer assessment works in your

situation.

• Discuss the question in bold on the slide. Note to Facilitator: Read these directions carefully to get to video segment: You will need to navigate the Annenberg Web page to get to the video and then advance the video to get to the segment. Select the link above to visit the Annenberg Learning Web site. From the Web page titled “Assessment in Math and Science: What’s the Point,” scroll down to “Workshop 7: That Would Never Work Here, Either!: Seeing Assessment Reform in Action, Part II.” Select the icon to open the video window. After the Annenberg introduction, select and drag the time marker to 1:07:48. The segment ends around 1:23:29. Video URL: http://www.learner.org/vod/vod_window.html?pid=1039

21

• Now we’re shifting gears to analyzing data in such a way that it is useful information for instructional practice.

22

• In this section, we will focus on analyzing data and taking action based on

data analysis.

• Item-level data is a gold mine for diagnosing student learning needs. We will

take five approaches to mining item-level data:

• Here are the five methods of analyzing data we’ll be learning.

• In general, the process is to start with looking at group data then digging into individual student data.

• We’ll also be looking at ways to analyze selected-response data as well as constructed-response data.

23

Note to Facilitator: Project this slide for participants to refer to avoid jumping back and forth as you move to subsequent slides (they may also refer to their copy of the PowerPoint handout). Point out features and answers to the questions that follow. • Here is an excerpt of a table where the test data has been sorted by the domain

and standard.

• It shows 3 test items that all assess the same domain and the same standard.

• Let’s assume that the rest of test items measured other domains and standards and are found below this excerpt.

• For simplicity’s sake, we are only looking at these three items.

24

Note to Facilitator: Project the table and mask off columns 8-12. Point out where the data is found for the bottom box [review questions on slide].

25

• Looking at the data, we see that 85 percent of the students answered #6

correctly and that this item had the highest percentage of correct answers of

the three items.

• This was about the same as the entire grade level and the district. • Turn to your elbow partner and answer the question under the table. Be prepared

to share out. Sample Response Which item had the lowest percentage of correct answers? How does this compare with the rest of the grade? The district? Item #1 had the lowest percentage correct. This was about the same for the grade level but much lower than the district percentage.

26

Note to Facilitator: Project the table and reveal columns 8-12. Point out where the data is found mentioned on the next slides. • The question that this part of table answers is: Which wrong answer was

selected most often?

27

Note to Facilitator: Show where the answer to the question is found in the table. • Looking at #1.a.; the correct answer was C, so we look at the data for

distractors A, B, and D.

• The distractor that was selected most often was D. What does this mean?

• When distractors (wrong answers) are written for the typical

misunderstanding and errors that students make, diagnosing learning needs

is easier.

• On the next slide is item 1a, the multiple-choice part of item 1 on the test.

• The distractors were written to diagnose common student errors.

• In parentheses are the misunderstandings that would lead to choosing each

distractor.

28

• Here is the multiple choice part of Item 1.

• The distractors were written to diagnose common student errors.

• In parentheses are the misunderstandings that would lead to choosing each

distractor.

• This approach is also referred to as error analysis Sample Response What student learning needs do you diagnose in students who selected D for item #1? The students need to learn that the perimeter is two times the length plus two times the width and under what conditions to multiply and to divide to calculate the missing dimension.

29

• Now we are going deeper from the group data into the data for individual students.

• When a learning need is diagnosed from looking at group data, the next

step is to dig down further and examine individual student data.

• Table 2 shows how to display the multiple choice responses made by

individual students for one item.

• Column C is shaded because it is the correct answer.

• This table answers the question: Which students have the learning need?

30

• 1.b. Is the second part of the question. It is a constructed-response item. Now we’re moving into analyzing data this type of data.

• This table shows how the data for constructed-response items can be displayed.

31

[refer to slide]

32

Note to Facilitator: Show where the data is found in the table. • Look at the data for item 1.b.

• Turn to your elbow partner and answer the question in the bottom box. Sample Response Based on the criterion of proficiency for Extended-Response items, what percentage of students would benefit from re-teaching? 80% of the students would benefit from re-teaching. (7+28+40+5 scored below 3 or left the question blank)

33

• As was done in Table 2, the next step would be to dig into the data for

individual students who scored 0–2.

• Look at the scores each student got for each of the rubric criteria.

• Which of the rubric criteria had the lowest scores?

• Later, we will use the Alternate Ranking Method for analyzing this type of

data and creating flexible student groups.

• At your table group, answer the question in the bottom box. Be prepared to share out.

Sample Response Compare the Constructed-Response data for 1b to the multiple-choice data. 50 percent of the students answered the multiple-choice item correctly, while 20 percent were able to explain their thinking. What’s the learning need? What might this imply about instruction? Students need the opportunity to learn how to explain their thinking behind their answers to math problems. Instruction is needed to associate oral and written language with math problem solving and deepen their understanding of the algorithms

34

• Now that we’ve diagnosed students’ learning needs, we need more information about exactly what the students need to know and be able to do to be proficient.

• We’re going to dig even deeper into the items to determine the skills and knowledge behind them.

• When analysis reveals frequently missed items in the same content

standard or cluster, the next step is to go deeper in the mine by

deconstructing those items into the knowledge, skills, and big ideas required

to correctly answer them.

35

• As you learned in Unit 4, content standards must be broken into smaller

building blocks that provide a road map for instruction and assessment.

• If educators have developed assessment items by unwrapping standards,

task analysis, and creating clear learning targets (as presented in Unit 4),

this work is done, and they need only to revisit it.

• Many times, however, educators use ready-made tests or develop

assessments from item banks.

• Table 4 shows how to organize information from deconstructing a number of

items.

• In this table you would take 1b, an item that many students scored below proficiency, and unwrap it.

36

• Once a number of frequently missed items measuring the same standard

have been deconstructed, educators can look for patterns.

• These probing questions may also suggest next steps for adjusting

instruction.

[refer to questions on slide]

37

• What other hazards have you encountered when you’re analyzing data?

38

• Let’s review the steps we took from looking at big picture data and digging down into deeper, more detailed information.

[Review points on slide]

39

40

• This is the most important part of the module because without acting on the data, what’s the point in collecting it?

41

[refer to slide]

42

• The first example, Guided Groups is an instructional adjustment that can be done

on a minute-by-minute basis.

• Alternate Ranking is useful for adjustments on a weekly or daily basis.

• Recall the cone graphic that provides a visual display of these assessment sources.

43

Note to Facilitator: Show video and discuss questions on slide. Video URL: http://successatthecore.com/teacher-development/featured-

video.aspx?v=38

44

• In the video, the ELA teacher created flexible groups during the lesson.

• Assessment-literate educators also use weekly assessment data formatively

to temporarily group low-performing students for re-teaching and high-

performing students for enhanced learning.

• End-of-unit assessment data can be used for establishing groups needing

targeted intervention.

• Annual data can also be used summatively to program students into

intensive intervention classes.

• Alternate Ranking is a simple and versatile way of analyzing data for either

formative or summative uses.

• Constructed- or selected-response data can be analyzed.

• It is a somewhat formal method of analysis to identify and group students

for differentiated instruction.

45

Note to Facilitator: Refer participants to the Participant Packet for the Alternate Ranking Handout. You may also give this assignment as homework for the participants to apply this to their own students’ data. • The purpose of Alternate Ranking is to help educators analyze student

performance and create flexible groups for differentiated instruction.

• The big idea is to rank the students in an alternating fashion from highest to

lowest performance to form temporary groups for targeted instruction.

• In this activity, use the data set provided in the handout.

46

• After completing the activity, answer the questions on the slide in small groups.

• Be prepared to share out.

47

• Now we’re moving on to how assessment results can be effectively communicated to students as feedback.

48

• Again, there is a substantial body of research supporting the impact that quality feedback has on increasing student achievement.

• In Unit 5, we read a quote from Black and Wiliam (2009) about the power of

interactive dialogue and feedback that moves learning forward.

• In this section, we will examine what makes feedback effective.

Note to Facilitator: Have participants discuss question on slide as time permits.

49

• Note some key characteristics of effective feedback provided by Popham: [refer to slide]

50

• Expanding on Popham’s definition, Jan Chappius points out six characteristics of effective feedback that focus on the QUALITY of the feedback itself. [refer to slide]

51

52

• Now let’s turn to how educators can improve the quality of their feedback by looking at some examples of effective vs. ineffective feedback.

• Turn to your elbow partner and match the ineffective feedback to its more effective counterpart.

• Discuss what makes the counterpart effective. Answers: 1B, 2D, 3A, 4C

53

54

• Now we’re shifting to talking about students in an active role in using their data.

• We learned in Unit 5 that assessment-literate educators involve students in

the assessment process as an essential part of balanced assessment.

• When students become partners in the learning process, they become

owners of their learning and gain a better sense of themselves as readers,

writers, and thinkers.

• When students reflect on what they have learned and on how they learn,

they can adjust their learning tactics and develop their own learning goals to

become more effective learners.

• In this section, we examine setting the stage for reflection and goal setting

and then an activity and tools.

55

• It takes deliberate work to build trust between students and teachers and among students to create an environment where students can honestly reflect on their learning.

• Things assessment-literate educators do to set the stage for student

reflection and goal setting include: [refer to slide]

56

Video URL: http://dww.ed.gov/Data-Driven-Instructional-Decision-

Making/Student-Use-of-

Data/see/index.cfm?T_ID=30&P_ID=80&c1=1570#cluster-1

57

Note to Facilitator: If you will be meeting with the group again, assign homework to bring the forms and templates they use to help students use their assessment data for goal setting. Show and Tell at your next gathering. What challenges were there? How did you meet the challenges? • In Seven Strategies of Assessment for Learning (Pearson 2009, 111-117),

Jan Chappuis provides steps to lead students to use formative assessment

results to set goals.

• Step 1: [refer to slide]

58

59

Note to Facilitator: Have participants explore these links during the workshop as time allows or encourage them to do so on their own time.

60

• The steps on this slide are a modified version of the goal setting steps for constructed-response items.

• Note the importance of student understanding of the rubric.

Discuss: • Has anyone tried using student-friendly rubrics as a tool for student goal setting?

What was your experience? Would you do it again? Why or why not?

61

• Earlier in this unit, we presented ways that assessment-literate educators

adjust instruction.

• As partners in the learning process, students can learn to adjust their

learning tactics.

• In alignment with the college- and career-readiness standards in the CA

CCSS and 21st Century Skills, schools should ideally prepare students to

be autonomous, self-correcting learners who continually monitor the

effectiveness of their tactics.

• The reality is that students must be willing to do this.

• The challenge for educators is to set up the conditions to lead students to

doing it.

• Actions that assessment-literate educators can take are: [refer to slide]

62

• At first, students who are not accustomed to reflection may give fuzzy or

shallow responses.

• For example, they may state their feelings about the work or give overly

general responses such as, “This was boring.” Or, “I need to work harder.” [Refer to slide] Discuss: • What other ways do you help students to go deeper in their reflection about their

learning progress?

63

64

[refer to slide] Note to Facilitator: Open up to questions as time allows.

65

• Enter your answers in your journal and discuss in small groups. Sample Responses: 1. Name and describe the three basic parts of a quality rubric. Criteria are “What counts” in a product or performance Scoring levels can be two or more levels of quality ranging from excellent to poor. Descriptors define the scoring levels. 2. Suppose you placed a student in a group to reteach a unit, based on your analysis of multiple-choice assessment item-level data. If a parent asked you why you did this to their child, explain your approach to item-level data analysis in simple terms. I purposely designed the wrong answers to diagnose typical errors that students make. I found a number of students who chose the same wrong answer, I created a small group and pulled those students aside to review the information because understanding this information is critical to doing well with the lessons that follow. 3. State four characteristics of effective feedback. All of the following characteristics are effective: -Given to students as quickly as possible in a useful format. -Treated as helpful indicators of what needs to be worked on.

66

-Descriptive and focus on areas of both strength and weakness. -Includes suggestions about ways students might address their weaknesses such as: • Focuses attention on the learning, not the student. • Occurs during learning, while there is still time to act on it. • Addresses partial understanding. • Does not do the thinking for the student; does not give total solutions or next

steps. • Limits corrective information to the amount of advice on which the student can

act. • Given in time to implement or act on. 4. How would you persuade a colleague to involve his or her students in using the results of assessment? What activity or strategy would you recommend as a starting point? I would point out the research showing relationship with increased achievement. I would recommend starting where the teacher is most comfortable in taking a risk.

66

Time: 10 minutes • Pease take a few minutes to complete the post-assessment in your participant

packet to gauge your knowledge of assessment literacy. Answer Key for Post-Assessment: 1. B 2. D 3. A 4. B 5. C 6. A 7. B 8. D 9. A 10. C

67