Innovative Assessment Systems in Action · Innovative Assessment Systems in Action •Washington’s work to implement use of Smarter Balanced interim assessments to connect to Digital

Innovative Assessment

Systems in Action

National Conference on Student AssessmentAustin, TX

June 30, 2017

Presenters

• Heidi KroogSmarter Balanced Assessment ConsortiumDirector, Supports for Instruction

• Anton JacksonWashington State Office of the Superintendent of Public InstructionMathematics Assessment Specialist

• Nancy Thomas PriceIdaho State Department of EducationComprehensive Assessment System Coordinator

• Gail PaganoSmarter Balanced Assessment ConsortiumSenior Director, Member Services

Presentation Outline

Topic PresenterTime

(min)

1. Overview of Smarter

Balanced Assessment System Heidi Kroog 10

2. Perspectives from

Washington State Anton Jackson 20

3. Case Studies from Idaho Nancy Thomas Price 20

4. Discussion Gail Pagano 10

The Smarter Balanced Assessment System

What’s New for Smarter Balanced?

• New Digital Library Storefront Site

What’s New for Smarter Balanced?

• Item Mapping & Connections Playlists

Washington and OSPI• Office of Superintendent of Public Instruction (OSPI)

www.k12.wa.us

• Anton Jackson-Mathematics Assessment [email protected]

• Shelley O’Dell-ELA Assessment [email protected]

6/28/2017

OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION

7

http://www.k12.wa.us/

mailto:[email protected]


Innovative Assessment Systems in Action• Washington’s work to implement use of Smarter

Balanced interim assessments to connect to Digital Library resources and to Smarter Balanced summative assessment student performance data

• State-level grant

• School district

• Middle school

6/28/2017


8

Washington Student Achievement Council (WSAC)

• 2015–16, Washington Student Achievement Council’s Educators for the 21st Century professional development grant program

• Focused on helping educators implement the Smarter Balanced interim assessments: http://www.wsac.wa.gov/21-educators

• Report: Hanley, Daniel (Western Washington University) & Ellen Matheny (2016). Educators for the 21st Century Statewide Evaluation Report: Implementing the Smarter Balanced Interim Assessments. Olympia, WA: Washington Student Achievement Council.

• Contact: Mark [email protected]

6/28/2017


9

http://www.wsac.wa.gov/21-educators


WSAC Grant Results• Funded four projects

• Across the projects:• 135 teachers administered either an IAB or ICA

• Administered IABs more frequently than ICAs

• Majority of teachers administered both IAB and ICA

• Number of times students took IAB or ICA varied greatly

• Administered ICA to: • See how prepared students were for the summative assessment

• Plan instruction

6/28/2017


10

Connections to Digital Library and Formative Assessment

• Educators increased ability to use interim data formatively to identify next instructional steps:

• Identify individual students needing additional support (most frequent)• Close achievement gaps between groups of students (less frequent)• Improve academic success of all students

• Timely interim results supported use of Digital Library resources• Administrators pre- to post-grant self-rating of ability to use Digital Library

resources increased from 1.87 to 2.6 (on 4-point scale)• Educators pre- to post-grant self-rating of ability to use Digital Library

resources increased from 1.49 to 2.65 (on 4-point scale)

6/28/2017


11

Educator Feedback on WSAC• Knowledge/skills, initially, low in scoring interims and using data to

inform instruction. By the end of the project, knowledge/skills were “moderately high.”

• By end of project, particularly high knowledge/skills:• Interpreting student and class reports• Using data at the classroom and school level to inform instruction

• Identified needs for continued professional development in:• Interpreting data • Hand scoring interims• Using Digital Library to support classroom instruction• Interpreting/using interim reporting categories (Below Standard, At/Near

Standard, Above Standard)

6/28/2017


12

Tumwater School District• Approximately 6500 students

• Approximately 375 classroom teachers

• 10 school buildings• 6 Elementary schools (grades Prekindergarten–6)

• 2 Middle schools (grades 7–8)

• 2 High Schools (grades 9–12 and alternative)

• Contact: Tara Richerson-Tumwater School [email protected]

6/28/2017


13


Tumwater: Interim Use• Administered 1709 ELA and 1233 math IABs• No formal organization

• Different buildings selected different approaches• Most IABs given in March and April

• Used two ELA IABs: Editing and Revision• All students in grades 7–10• Twice a year (pre- and post-) as benchmarks

• Middle school special education teachers used IABs from lower grade levels to gather information about student performance at student’s ability level

6/28/2017


14

Tumwater: Goals• Provide students some testing experience

• Prepare for summative testing• Focused on ensuring students felt comfortable with test tools and item types

• Provide teachers data exposure and interpretation practice• No specific protocol used• Elementary teachers have used item analysis for some time with other

assessments, but was new for high school teachers• High school teachers used item analysis component available in AIR Ways• Get into the data for how to fill in gaps in what they know about students• Focused on interpreting data and consider them when planning next steps

• Focus of secondary ELA on common assessments to support PLC conversations

6/28/2017


15

Tumwater: Educator Supports andFormative Assessment

• Two instructional coaches (at elementary level) provided trainings using the IABs, including reviewing and using data in AIR Ways for instructional next steps

• ELA TOSA (at secondary level) supported ELA teachers using data and providing interventions for struggling students

• Training delivered over three months, three days, 1 hour each day• First two meetings face-to-face and the third virtual (Google Hangout)• February: Testing tools support, teaching for high-yield results to extended response items• March: Vocabulary of SBA, Performance Task talk, writing prompt format for SBA-like

practice• April: Final push (how not to burnout), interactive review games, and lessons learned

• Success through bumping up FTE for student information: extra staffing facilitated getting records into system easier and made testing available smoother

6/28/2017


16

Tumwater: Comparing Interim Types• Appeal of the IABs over the ICA?

• Simply, IABs shorter than ICAs• Most IABs not require hand-scoring• IABs: reliable, valid, align to specific learning• IABs useful for the TPEP goals around student growth

• No use of the ICA in 2015–16

• Only one high school math teacher has used ICA in 2016–17• Intent to predict 10th graders success on summative SBA this year

• In terms of ELA vs. math• ELA used more than math because everyone K–12 works with ELA standards• Social studies teachers and librarians can use ELA Research IABs• Science teachers can use ELA Informational Text IABs• Math has a more discrete audience: math teachers

6/28/2017


17

Brewster Middle School• Within Brewster School District

• Approximately 1000 students• Approximately 65 classroom teachers• 4 school buildings

• 1 Elementary school (grades Kindergarten–5) • 1 Jr/Sr High school (grades 6–12)• 1 alternative High school

• Brewster Middle School• Approximately 200 students• Approximately 15 classroom teachers

• Contact: Greg Austin-Brewster Middle [email protected]

6/28/2017


18


Brewster: First Steps• Previously tried options to evaluate student progress: hand-scored interims, MAPS,

unit assessments

• Decided to use interims 2 ½ years ago to monitor student progress and inform instruction• Went strictly with IABs last year per district superintendent: Top down commitment was a huge first step• Helped focus on one main form of assessment• Given basic introduction on implementation: principal had some trainings on using IABs and available data• Data analysis coordinator worked to become familiar with the IAB process and shared with principal

• Whole staff training on a late start Monday lead by specialist

• PLCs• Several ELA PLCs team trainings with specialist support• Covered interim usage and hand scoring in math PLC group

• A lot of the work involved trial and error• Getting in the IAB's and “getting my hand dirty,” trying to figure it out “on my own”• Look at items, hand scoring resources, and interims as a whole

6/28/2017


19

Brewster: Steps• Aligned IAB testing calendar: which IABs to give and when to create unity

• Principal set scoring teams

• Collaborative scoring process• Went through hand scoring materials together to ensure consistency of scoring• Several late start Mondays and collaboration time• Math, science, ELA, PE, etc. teachers all looking at student writing and scoring rubric• Approach was successful; whole grade band saw what happened within the IAB work

• Takes some forward thinkers to really push the IAB use in the classroom

• Used resources available through the WCAP site including the Digital Library and formative assessment practices to guide post-interim instruction

6/28/2017


20

Brewster: IAB Administration• Varied by teacher, with guidance

• Principal laid out timeline expectations on administering IAB's• Educators selected IAB based on what best lined up with their instruction• For ELA, began the year with the Language and Vocabulary Use IAB

• All staff assisted with scoring the Brief Writes IAB• Justification: all content area teachers teach reading/writing skills• Further justification: student success on summative assessment very dependent on writing skills• Worked well having staff support and be supported by ELA teachers

• Success factors• Coordinator who is motivated to help and learn, eager to share the latest and greatest• Chromebooks: 1:1 made administration easy; no longer at the mercy of "the cart schedule“• Team scoring and data evaluation

• Seeing both what students are missing but also on what they are getting

• Less about student performance and more about student progress, trends, areas of understanding, areas of student confusion

6/28/2017


21

Brewster: Questions and Rubrics• Question types on IABs

• Knowing how questions are asked on IABs changed/shaped questioning on classroom assessments

• Helped build curriculum that mirrors the types of questions/tasks students see on summative assessment

• Rubrics helped classroom expectation development• Students are given rubric

• First break down as a class, then provided examples to score, finally students go to writing• Backward to how writing is usually taught; typically students don’t see rubric or first see the

rubric when getting final writing back, scored and/or graded• Push teachers to think about how to use writing rubrics to drive student writing

• Increase awareness of what “grade-level” looks like• Helped build classroom rubrics that mirror expectations of summative assessment

6/28/2017


22

Brewster: Using Data to Inform Instruction

• AIR Ways Reporting system• Looking at students responses to individual IAB questions: formative practices of

interpreting evidence then acting on evidence for upcoming units of instruction• Identify specific standard(s) with which students need more practice

• Using evidence to support inferences • Determining theme/central idea• Identifying text structure

• Using data to find review, strength, weakness areas• Previously: compare student growth from last year, or to last years’ students, or to

state data• Now: finding where kids are right now; making that the data of focus

• Question-type issues students may have• Reading strategies so students can decipher what the question is asking• Considerations of resources to support student learning

6/28/2017


23

Brewster: Pitfalls• Took on too much at the beginning

• Did the Brief Writes IAB in one day; overwhelming for students• Jumping into interim without previewing the items; can be too much for students depending on time of year

• Hand scoring is a challenge• Time to hand score, to get teacher buy in• Hand scoring a huge benefit to ALL teachers of all subject areas; see expectations of grade-level writing and

where students are

• Holding ALL teachers accountable to the IAB work• Communicating student scores; letting students know how they did and what they can improve• At first, interim data wasn’t helpful for individual student instruction or teacher planning• Now those “kinks” are sorted out for the most part; information is much more helpful

• Hardest part: getting students to give IAB their best shot• One class last year was highly intrinsically motivated• This year, had to attach rewards to their performance to get more students to give their best effort• Data use increased student buy in

6/28/2017


24

SHERRI YBARRA, SUPERINTENDENT OF PUBLIC INSTRUCTION

SUPPORTING SCHOOLS AND STUDENTS TO ACHIEVE

Because…this is where change happens

This system was designed to

support teachers with student

mastery of state standards.



Today…

How widely are assessment system components being used?

How does the system inform teaching and learning?

What does implementation look like?

What are the results? Case Studies



Statistics about use of Interim Assessment

District Implementation Types** Number of

Districts 2016

Number of

Districts 2017

Systemic Users 3 5

Deliberate Users 20 18

Intentional Users 37 33

Emerging Users 16 29

Curious Users 33 44

Total 109 131*

* Idaho has 159 school districts, 41 of which are Charter LEAs

** Implementation Types based on number of different assessments used, years involvement,

grade level and school usage, data usage as indicated by large scale improvement in

summative proficiency



Use of system components

2016 2017

Interim

Assessments

197,500 295,693

Digital Library 2412 Users 2968 cumulative

(approx. 15K teachers)

• Air Ways: New Interim Assessment data tool shows

top 5 (correct) items and lowest 5 (incorrect) items –

AIR

• Digital Library Connections Playlists



Informing teaching and learning

Examine summative data of

incoming students in the Fall

Determine if / what additional

detail is needed

Administer Interim Block

assessments from previous

grade

Examine class, student, and

item details from IABs

Reference Item Specifications

task models

Identify supplemental Digital

Library tasks / activities for

formative use



Informing teaching and learningClassroom Claim and Assessment Target Reports



Air Ways Interim Assessment Data Tool

Top five and lowest five performing items



Digital Library

Connections

Playlists

Connections to

useful

instructional

support activities

based on Interim

performance



Implementation Intentionality

• Which assessments will be administered based on goals or

questions to be answered?

• Which students will participate in the assessments?

• When will the assessments be given?

• Will assessments be given on grade level or at different grade

levels?

• Where will assessments be administered and by whom?

• How will the data be used and with whom will it be shared?

A well-thought out, written and agreed upon way to

proceed:



Implementation Stages

Logistics of giving Interim Assessments:

Teachers learn how to proctor tests

Teachers learn how to access and read reports

Manage rosters / student groups effectively (data set)

Make sense of results/data:

Collaboration Time / PLCs

Use interactive capability of reports

Student item error analysis (Air Ways)

Change instruction; address misconceptions

Further depth of knowledge questioning on a day-to-day basis

Use task models

* Interim Assessment Implementation Planning Checklist, idaho.portal.airast.org



Implementation Steps

* Interim Assessment Implementation Planning Checklist, idaho.portal.airast.org



Outcomes: Benefits to Students

• Students competence with software and tools“I believe teachers value students being familiar with the test delivery system.”

Renee, Idaho Falls School District

• Practice with various accommodations and

designated supports Pocatello School District

• Students and teachers become more

familiar with item types “I never asked my students to do that!”

Andi, Idaho Arts Charter School

• Reduced anxiety on the end of year assessment“Our students said it made all the difference in the world.”

Georgeanne, Lakeland School District Assessment Coordinator

• Reduction in summative testing time“Students felt more prepared, more relaxed, and more committed to doing

well. It also cut our testing time on ISAT way down.” Sherry, Marsh Valley

• Improved student ownership of learning “Dad, look how good I did, I was

always afraid of these tests before.” Mountain Home student



Outcomes: Benefits to Teachers

Affect instruction while there is still time to change the outcome.

“I am so excited to have the IABs to help me pinpoint areas my students

need to work on. I discovered with one of the 5 most missed questions, that is

was a simple matter of making sure they read the question carefully. I know

also know that many of them need work relating fractions to decimals. Such

valuable information to have!”

Sue D. West Ada teacher

“I cannot tell you how hopeful I am, looking at the data on the Air Ways

interface. I feel like the blinders have been removed. I can see where our

strengths are and I can make plans that address our weaknesses. This is a

powerful tool.”

Julie B. West Ada teacher

Lava Elementary School



• One of four elementary schools in the

Marsh Valley School District, Southeast Idaho

• Combined classroom of 27 students (Gr. 3 and 4)

• Math, Third grade 100% proficient for 2 years, 67% Level 4..

Math, Gr. 4, 94% proficient this year, 72% Level 4.

ELA/Literacy Gr. 3 78% proficient with 56% Level 4.

ELA/Literacy Gr. 4 89% proficient with 72% Level 4.

• Teacher attributes her success to use of Interim Assessment data in

combination with use of the Item Specifications task models.

“Thank you so much for the email and medal! Haha..Yes, it does take so

much work, but so rewarding. I am so happy. I cried and was emotional for 3

days. That probably sounds weird, but after so much time and effort, it was an

overwhelming feeling.” Tanny Davids

Marsing High School / Marsing School District



• Demographics: 833 students in 3 schools

• Marsing School District 67% low income, High School 62%

• Initial training in Marsing School District was September 1, 2016

• Follow up training included teachers in November 11, 2016

• Data from Interim blocks reviewed as well as Digital Library and other

instructional resources.

• SDE Content Coordinators participated for instructional focus

2015 2016 2017

Gr. 10 ELA/Literacy 70% 71% 90% proficient!

Gr. 10 Mathematics 25% 29% 73%

All grades ELA/Lit 42% 39% 47%

All grades Math 26% 27% 34%

• Grade 6 ELA/Literacy 16% greater proficiency

• Grade 7 Math 18% greater proficiency

Marsing High School / Marsing School District



“The staff has worked hard this year at helping our students improve and

it's paying off. We are using this time to reflect on the results and

looking for ways to continue to improve.”

Superintendent. Norm Stewart, Marsing School District

“…the use of the Interim Assessments has been the key for most of the

pockets of good growth! I am a believer and am spreading that word

wherever I go!”

Leadership Capacity Builder, Sue Beitia



Demographics

Schools: 4 HS, 4 MS, 13 Elementary Low Income 48%

Students: 12,840 Idaho’s 4th largest school district

Year 1 (2014-15)

• Pilot of Interim Assessment Blocks recommended by Comprehensive

Assessment Committee

• Focus: Teachers and Students become familiar with item types and test

interface

Year 2: (2015-16)

• Implemented IAB and embedded in units of instruction (district curriculum)

• Focus: Students become familiar with tools and test settings and teachers

understand how standards are assessed.

Year 3: (2016-17)

• Teachers use data to inform instruction!

• Use of the Teacher Hand Scoring System and Air Ways reporting

Lessons Learned:

• Make the IABs easily accessible

• District upload users/rosters

• Increase access to technology

• Provide ongoing district PD and in-

building support

• Monitor the Idaho Portal for updates

• Celebrate successes

• Kathy Luras, District Curriculum/Assessment

• [email protected]

Consistently above state averages and improvement percentages(The Idaho Standards Achievement Test by Smarter Balanced)



Best Practices

Districts that made the largest gains in percentage of

proficient students on year end assessment have these

practices in common:

• Strong leadership to support implementation

• Use of the entire system: Summative, Interim and Digital

Library (formative)

• Use of Interim Assessment thoughtfully

• Use of data to look forward and determine next steps

• Stay the course; improve implementation each year



What the data shows…

Summative assessment increasesState Data Districts Using Interim Assessment

Average % Proficient Improvement Per Grade

2.3% 6.0 Small District

7.3 Small District

9.5 Small District

3.0 Medium District

3.9 Medium District

1.7 Large District

4.9 Large District

5.4 Large District

Systemic implementation with targeted focus, school or district-wide agreed upon strategies

rather than each individual teacher use, blocks vs. comprehensive seems to yield greater

improvement.

Continuing needs across the state



• Understand a system of assessment

• Further develop solid data use and formative

assessment practices

• Improve overall assessment literacy

…. these are necessary “ understandings” to

maximize the resources available in our

comprehensive assessment system from Smarter

Balanced…

Nancy Thomas Price, Comprehensive

Assessment System Coordinator

Idaho State Department of Education

[email protected]

208-332-6988


Documents

Innovative Assessment Systems in Action · Innovative Assessment Systems in Action •Washington’s work to implement use of Smarter Balanced interim assessments to connect to Digital