Smarter Balanced Assessment review report

Smarter Balanced Assessment Report EXECUTIVE SUMMARY

Purpose of Report: To inform the School Board of the Smarter Balanced Assessment administration during the 2013-14 school year.

This spring, Sioux Falls School District students in grades 3-8 and 11 were among the 3 million students in 20,000 schools across the country to participate in the Smarter Balance Assessment pilot. Scores are not being reported, as this years test was conducted to ensure accuracy and fairness of questions in preparation of full implementation during the 2014-15 school year.

The test is aligned to South Dakotas new academic standards in English Language Arts and mathematics. Schools were asked to submit the successes of the pilot assessment, as well as the areas in need of improvement. While the Districts technology and infrastructure performed better than expected, there were some challenges with the actual assessment. The following detailed report has been submitted to the South Dakota Department of Education for further review and consideration.

Administrative Recommendation to School Board: Acknowledge the review of the Smarter Balanced Assessment administration 2013-14 school year.

Report Prepared by: Curriculum Services and ATI May 12, 2014 Presented by: Sharon Schueler

Smarter Balanced Assessment Review Report

Purpose of Report: To provide the School Board with a review of the Smarter Balanced Assessment administration 2013-14 school year. District Priority Area: Student Achievement Goal/Strategy: The District will provide multiple programs and pathways to support improved student achievement across the curriculum. Explanation: During the spring of 2014, approximately three million students in 20,000 schools across the Smarter Balanced Consortiums 22 governing states, including students in grades 3-8 and 11 in South Dakota, field tested a new online assessment which is called Smarter Balanced Assessment (SBA). During this first year of the new test which is aligned to South Dakotas new academic standards in English Language Arts and mathematics, the results will be used solely to ensure that test questions are accurate and fair for all students moving forward. No data on individual students will be available. In subsequent years, student-level data will be available to parents, as well as teachers, schools, and districts. South Dakota has been part of a group of states working together to develop the Smarter Balanced assessment. By field testing the new assessment, South Dakota did not have to test students twice during this transition year once on Dakota STEP and once on Smarter Balanced. It also gave teachers and students an opportunity to gain exposure to the new assessment and testing in an online environment. One of the goals of the field test was to uncover challenges before the full launch of the assessment system in the 2014-15 school year. The SD DOE anticipated that there would be challenges making this move, which is precisely why the field test was administered to ensure that the new assessment will be fully operational for all students. It also gave teachers and schools a chance to practice test administration procedures and students the opportunity to experience the new interactive test items and tools. Test Preparation:

The Districts assessment manager worked primarily on the preparation and implementation of the online SBA from March 3 through April 17, preparing all end users with instructions for the test administration, fielding questions that needed to be resolved, preloading school rosters for the test administration and for test accommodations, and contacting the SBA Help Desk.

Report Prepared by: Curriculum Services and ATI - 1 - May 12, 2014 Presented by: Sharon Schueler

Each school had a test coordinator who assisted in establishing the test schedule and preparing classroom teachers for the test administration. The school test coordinators time ranged from 10 to 30 hours of preparation and planning. Test coordinators at each school were responsible for entering each student receiving test accommodations along with the specific accommodations into a spreadsheet for uploading by the District.

Each classroom test administrator set aside from three to ten hours to read

documents and view Smarter Balanced required test administration modules/videos.

Test coordinators and principals commented that they look forward to continued discussion about how they can streamline the training process. Test Administration:

When checking the Chromebook functionality, from February 20, 2014, to the first day of the testing window, ATI reported that 200+ Chromebooks did not have the AIR (American Institutes for Research) Secure App. Over 25% of the devices were not functioning because the Chromebooks were logged in with an "owner." The remainder were fixed by following these three steps:

(a) verified the Chromebook was enrolled in the K12 domain (b) verified the Chromebook was in the SFPublic suborganization (c) forced a sync with the domain by launching the SDAP Public Session.

There were 155 Chromebooks out of 18,000 issued that were not functioning for

the Smarter Balanced test administration. Fifty-eight had been placed in developer mode (student configured the device) and another 97 had problematic serial numbers (duplicated on the K12 domain, upper/lower case mismatch, serial number on system not matching the tag on the case, etc.). Students used alternate devices while the building computer specialists collected these devices. ATI worked with the State to get them manually corrected and placed into the correct domain/suborganization.

The opening of the testing window was delayed one week as a result of SBA issues. On the first day of the testing window (March 25), one issue that occurred across the district (and it was not limited to Sioux Falls) was that students did not have the Non-Performance Task tests for Math and ELA on the Select a Test screen. Instead of having all four tests to choose from, test administrators were getting the two Performance Task tests only (Math and ELA). Smarter Balanced became aware of this issue, and it was escalated and resolved.

At 7:30 a.m. on April 1, (when 17 schools in the District had planned to

administer the SBA), the SBA Help Desk received calls from users across the country unable to log into the TA (Test Administrator) interface and TIDE (Test


Information Distribution Engine). AIR confirmed that users throughout the United States were unable to access the system. At 7:50 a.m., a message was delivered to all users because of the inability to access the TA interface, TIDE, ORS (Online Reporting System). At 8:30 a.m., AIR resolved the issue.

Teachers felt much more comfortable and confident with the logging on process

once they had built their background knowledge through experience since the TA Interface was not available until the test was actually being administered.

When taking the test, some students were not allowed to go beyond a certain question regardless of what was triedcursor kept spinning; thus, they were not able to complete the test without shutting down and logging in again. Some students had difficulty with their confirmation code being accepted when trying to log in to a test session. Issues with not being able to move forward within a test or being thrown out of the test were resolved by having the student log out and back in again.

Another large issue that the District is continuing to deal with is students choosing the incorrect assessment to complete. If a student clicked on the wrong assessment from the choices listed, the expiration period started at that point, regardless if the teacher approved or denied access to the test. The Performance Tests (PT) expired within 10 days while the Non-PT tests expired within 45 days. The Non-PT was not an issue; however, we have had several students with expired PT tests.

In comparison to the estimated testing times for the previous Dakota STEP test in reading (2.5 to 3 hours) and mathematics (2.5 to 3 hours), the Smarter Balanced Assessments in ELA (4 to 4.5 hours) and mathematics (3 to 4 hours) were predicted to require an additional 2 to 2.5 hours to be scheduled for testing. The actual test administration averaged approximately 3 to 5 hours for each content area with many schools stating that the test administration time was shorter than anticipated. According to the April 23, 2014, publication of Education Week the operational test for Smarter Balanced in 2015 will be 7 to 8.5 hours.

Because of the preplanning completed by the Districts technology support team, overall, schools shared that the Districts infrastructure and Chromebook technology worked more smoothly than they had anticipated. Test Questions:

Students were not aware of how highlighting and selecting worked on certain test questions. Only certain words were options for the answer selection. The mouse pointer did not always change to a pointed finger to indicate a possible answer choice. This added to the difficulty in answering the question sufficiently to move on to the next page. Better directions are needed so students know where and how to highlight responses.


When the embedded calculator was accessed, it overlaid on top of the actual

math question. Students were unable to use the calculator and see the math problem at the same time. The embedded calculator was not user friendly compared to the actual calculators students had used in their math classes. After using an embedded notepad, students were unable to access the notepad and keyboard their response at the same time.

The text-to-speech was used for students requiring this accommodation; the

actual speech was difficult to listen to because of voice quality (robotic) and the buffering of the voice. As a result, some students chose to not utilize this feature. When the text-to-speech was not available as an accommodation, AIR confirmed that they believed the text-to-speech feature may have been incorrectly set in TIDE.

Adjusting the volume for the ELA test became an issue if the volume was not set before logging on to the Smarter Balanced Assessment. Some words were mispronounced by the computer, such as read; the voice would say go back to the passage you read (which should sound like red but was pronounced as reed).

When students were given the stacked translation accommodation, students

were to receive test questions with English and the Spanish translation stacked above the English translation. Students received test questions with English and Spanish for some questions; while others had Spanish only with no English. While trying to resolve this issue with the SBAC Help Desk, three days of testing were lost for some students.

Students at all grade levels (especially at Grade 3) experienced frustration with

responses that required extended keyboarding skills. Teachers reported students shortening their responses on constructed-response items because they lacked fluency with the keyboard.

Other Concerns:

A more useful report should be available so schools can monitor students who still need to makeup tests and can calculate more accurate completion percentages for participation rates. Test administrators struggled to monitor students on the TA interface because the students state ID numbers were used without any first and/or last names.

Requests were received from parents to remove certain questions/passages of

concern on the practice tests. The District does not have authority to alter the practice tests.


Requests were received from twelve parents/guardians to have their student(s) not take the Smarter Balanced field test because of personal beliefs. They were informed that all public school districts are required to administer the statewide assessment to all students in the tested grades, and all those students are required to take the assessment (South Dakota Codified Law 13-3-55; Administrative Rule of South Dakota 24:55:07:01). Principals shared that teachers observed five of these students randomly answering questions or choosing the same answer and completing the test in half the time of the other students.

Parents and teachers have asked whether or not South Dakota has plans to

provide the Smarter Balanced benchmark/interim assessments to districts for implementation.

When the Smarter Balanced Assessment becomes a computer-adaptive test in

2015, computer adaptive testing adjusts to a students ability by basing the difficulty of future questions on previous answers, providing more accurate measurement of student achievement, particularly for high and low-performing students. Some parents have asked how and if the structure of the test questions will address grade-level or above grade-level standards and how results will be reported.

Online assessments required students to have certain digital skills: using a mouse, highlighting text, dropping and dragging, drawing lines and creating graphs on a screen, operating an online calculator, using scroll bars, and keyboarding, to name a few. If online assessments are to capture a student's level of learning, rather than that student's ability to navigate technology, teachers must continue to integrate these skills into their instruction, giving students opportunities to build a level of comfort and confidence. Moving from a knowledge/comprehension-based assessment to a system that asks students to think critically and apply their knowledge means students will be better prepared not only for postsecondary education and careers but also for life. Principals and teachers commented that they look forward to continued discussion and collaboration on the transferring student learning from classroom instruction and student performance on formative assessments to the online assessment environment. Committee Participation: A committee was not formed; however, feedback was collected from building principals, test coordinators, and test administrators. Anecdotal feedback from parents is also noted. Costs: NA Summary: One of the goals of the field test was to uncover challenges before the full launch of the assessment system in the 2014-15 school year. The SD DOE anticipated that there would be challenges making this move, which is precisely why the field test was


administered to ensure that the new assessment will be fully operational for all students with feedback from all end users. The field test also gave teachers and schools a chance to practice test administration procedures and students the opportunity to experience the new interactive test items and tools. This report will be shared with the SD DOE. Administrative Recommendation to School Board: Acknowledge the review of the Smarter Balanced Assessment administration 2013-14 school year.


ES Smarter Balance AssessmentExecutive Summary

Smarter Balanced Review -- After Cabinet

Documents

Smarter Balanced Assessment review report