When You Wish Upon A Star: Computer-Based Assessments for NAEP
Overview: Computer-Based Assessments for NAEPMary CrovoNational Assessment Governing Board
Policy and measurement goalsImproving the form and use of NAEPCBA for all new and updated NAEP frameworksBoards Schedule of NAEP Assessments focuses on CBAMaintaining NAEPs gold standardAdvancing innovative assessmentsTesting complex content and skillsCapturing a range of student responsesIncreasing student engagementProviding greater accessibility for students
Focus areas for audience feedbackSharing lessons learned with states and districtsSchool-based IT challengesPushing the measurement envelopeInnovative reporting strategies*
Designing and Developing Computer-Based Assessments (CBA)in NAEPJay CampbellEducational Testing Service
The NAEP CBA TimelineTechnology-Based Special Studies WritingMathProblem-SolvingScience Interactive Computer TasksWriting8th grade12th grade
Math Adaptive Study8th grade
Writing4th gradeTech and Engineering Literacy8th grade*
Use of Computer Technologies to Enhance AssessmentExpand constructs measured and contexts representedTechnology-enhanced constructs (e.g., writing, problem solving)Simulated environments and extend contexts in which student can perform skillsScenario-based tasks allow for representation of more authentic and complex situations
Use of Computer Technologies to Enhance AssessmentCapturing more than just student answersActions in completing tasks reveal processes engaged by studentsExamine which tools are used by students and how they are usedRecord of time spent on task could be an indicator of engagement with task*
Use of Computer Technologies to Enhance AssessmentAdaptive TestingAt either the individual item or test section levelProvides for more precise targeting of content to student proficiency levelIncrease reliability of scores across performance distributionAchieve efficiency in testing timeLikely making testing experience more positive*
Use of Computer Technologies to Enhance AssessmentEngagement and MotivationSitting in front of computer likely more engaging than responding on paperEmploying cognitive principles of engagement and gaming design can enhance motivationIncreasingly more authentic an experience for many activities
Lessons Learned to DateProcessesPeople*
Lessons Learned to DateProcessesTime and expense requires more phased and deliberate approachMore integrative and iterative than with paper-and-pencilEvidence-Centered DesignModel basedEvidence focusedDriven by reporting goals*
Lessons Learned to DatePeopleSubject matter specialistsCognitive and learning scientistsPsychometrics and data analystsInterface and task designersGaming expertsSoftware and hardware specialists*
Administering NAEP Computer-Based AssessmentsDianne WalshWestat
How is the WCBA Administered?
Schools provideRoomDesks or tablesElectrical outletsNAEP providesLaptopsEarbudsMouse/mousepads*
How is the WCBA Administered? Before students arrive NAEP staff unpack and set-up 15 workstations and one Admin computer. Student laptops are connected to Admin laptop via a wireless router.
How is the WCBA Administered?
During the assessment.Students enter 10-digit ID to log in to the system.
NAEP staff monitors students progress on the Admin laptop.*
How is the WCBA Administered? After the assessment.NAEP staff connects the Admin laptop to the Internet and transmits student data.
NAEP staff connects the Admin laptop to the Internet and transmits student data.*
How are students with disabilities and English-language learners assessed?Universal Design: features that are available to all students. These features are integrated into the system, and all students elect whether/when to use them.
Accommodations: offered only to students who require them (SD or ELL). If provided through the computer, these features must be enabled.*
How are students with disabilities and English-language learners assessed?Universal design elements on recent writing computer-based assessment include: Small group/one-on-one Text to speech Adjusting font size up to 48 pointsAdjusting contrast/colorsUse of highlighterSpell-check
How are SD and ELL students assessed?Accommodation elements on recent writing computer-based assessment include: Extended timeMagnification greater than 48 point fontDirections in SpanishBreaks during testTesting in a separate sessionOther
How was Universal Design received by schools and students?Many positive comments about being able to include students without singling them out.20% fewer separate accommodation sessions in CBA then in paper-and-pencil assessments.Less burden on schools to find space and staff to administer accommodations.Students accommodations were less obvious to other students.Students were able to access accommodations as they chose.*
How was the WCBA received by schools and students?Students were engaged in schools in varying demographics (setting, SES, etc.)Principals and teachers were excited to have their schools participate.Students used tools with ease.
Computer-Based Assessment of Technology and Engineering Literacy (TEL)Malcolm BauerEducational Testing Service
OutlineTechnology and Engineering Literacy (TEL) AssessmentWhy Computer-Based Assessment (CBA)?Innovations that CBA supportsReportingConstruct definitionScoringTask Development*
TEL Assessmentthe capacity to use, understand, and evaluate technology
2010Student/evidence model refinement2011Task model and task development 2012Series of tryouts/pilots for task revision Large-scale psychometric pilot test Probe assessmentTEL Milestones*
Innovation in ReportingProvide detailed, in-depth reporting of student competenciesProvide information about cognitive processes and strategiesMake results more actionable better able to inform policy and practiceReport these enhanced results in ways that are more informative and useful*
Innovation in Process: Evidence-Centered DesignStudent ModelWhat you want to measureHow to recognize and interpret observable evidenceHow to elicit good evidence of what you want to measureCan students design or redesign a device or system to address a need?Designs are evidence of students ability.
Look to see whether students satisfied requirements.Have students design something.
Can they identify and meet the needs (requirements)?*
Innovation in ScoringEvidence of new competenciesModel-driven selected responseEvidence-rich work productsEvidence of cognitive processesSame as abovePattern tracking*
Item Types: Scenario-based Task and Discrete ItemsTEL Domain*
Innovation in Task DevelopmentScenario-based tasksAre extended assessment activities which combine multiple measures of student abilityLeverage computer delivery to offer innovative, interactive assessment tools*
Summary: Uses of CBA in TEL AssessmentAllow students to perform educationally relevant and engaging activitiesCollect a variety of evidence of TEL competency and processScore student responses to tasks and interactions with tasksProvide access to TEL results that are informative and useful*
AcknowledgementsNational Center for Education StatisticsNational Assessment Governing BoardAll members of the TEL ECD Team, the TEL Task Development Team, and Measurement TeamHager Sharp Inc.*
NAEP staff bring 15 student laptops plus 1 admin laptop the NAEP staff use to monitor to the assessment 1 wireless router Computer accessories (ear buds, mice, mouse pads) Assessment materials (Planning Papers, writing brochures, pencils)
Schools only need to provide a room, desks or tables for the laptops, and electrical outlets. Because the assessment is localized on the students laptops, school do not need to provide Internet access.
*Before students arrive, NAEP staff unpack and set up 15 student laptops and accompanying material. Once the laptops are set-up, staff plug in and turn on the wireless router.Then the admin application is launched from the laptop the NAEP staff use.The student laptops are now connected to the admin laptop and the assessment is ready to start.
*Two groups of 15 students are assessed. Students enter a 10-digit ID number, then begin the assessment.The timing and all instructions are built into the assessment application. The assessment takes about 2 hours to complete.While students are taking the assessment, their progress is monitored on the NAEP staffs Admin laptop. Student data is transmitted to the Admin laptop via the router throughout the assessment.
*After the assessment, laptops and other equipment are packed up.NAEP staff then connect the Admin laptop to the Internet, and transmit the data to a central server for scoring.*The assessment was designed with a number of universal design settings. This means there are a number of tools build in that anyone can access, not just students identified as SD or ELL.
This includes small group or one-on-one because of the nature of the assessment. Student can also adjust the font size from 14 to 48 point. There is a speak function that can read aloud all or selected portions of directions, tasks, and stimulus materialsThere is a choice of background colors, including a high contrast option.A highlighter students can use for writing pro