When You Wish Upon A Star: Computer-Based Assessments for NAEP

Overview: Computer-Based Assessments for NAEP

Mary CrovoNational Assessment Governing Board

Policy and measurement goals

• Improving the form and use of NAEP• CBA for all new and updated NAEP frameworks• Board’s Schedule of NAEP Assessments focuses

on CBA

• Maintaining NAEP’s “gold standard”• Advancing innovative assessments• Testing complex content and skills• Capturing a range of student “responses”

• Increasing student engagement• Providing greater accessibility for students

3

Focus areas for audience feedback

• Sharing lessons learned with states and districts

• School-based IT challenges• Pushing the measurement envelope• Innovative reporting strategies

4

Designing and Developing Computer-Based Assessments (CBA)in NAEP

Jay CampbellEducational Testing Service

The NAEP CBA Timeline

Technology-Based Special

StudiesWritingMath

Problem-Solving

ScienceInteractive

Computer Tasks

Writing8th grade

12th grade

Math Adaptive Study

8th grade

Writing4th grade

Tech and Engineering

Literacy8th grade

6

Use of Computer Technologies to Enhance Assessment

Expand constructs measured and contexts represented• Technology-enhanced constructs (e.g.,

writing, problem solving)• Simulated environments and extend

contexts in which student can perform skills

• Scenario-based tasks allow for representation of more authentic and complex situations

7

Use of Computer Technologies to Enhance Assessment

Capturing more than just student answers

• Actions in completing tasks reveal processes engaged by students

• Examine which tools are used by students and how they are used

• Record of time spent on task could be an indicator of engagement with task

8

Use of Computer Technologies to Enhance Assessment

Adaptive Testing• At either the individual item or test

section level• Provides for more precise targeting of

content to student proficiency level• Increase reliability of scores across

performance distribution• Achieve efficiency in testing time• Likely making testing experience more

positive

9

Use of Computer Technologies to Enhance Assessment

Engagement and Motivation• Sitting in front of computer likely more

engaging than responding on paper• Employing cognitive principles of

engagement and gaming design can enhance motivation

• Increasingly more authentic an experience for many activities

10

Lessons Learned to Date

Processes

People

11

Lessons Learned to Date

Processes• Time and expense requires more phased

and deliberate approach• More integrative and iterative than with

paper-and-pencil• Evidence-Centered Design

• Model based• Evidence focused• Driven by reporting goals

12

Lessons Learned to Date

People• Subject matter specialists• Cognitive and learning scientists• Psychometrics and data analysts• Interface and task designers• Gaming experts• Software and hardware specialists

13

Administering NAEP Computer-Based Assessments

Dianne WalshWestat

How is the WCBA Administered?

Schools provide…• Room• Desks or tables• Electrical outlets

NAEP provides…• Laptops• Earbuds• Mouse/mousepads

15

How is the WCBA Administered?

• Before students arrive…• NAEP staff unpack

and set-up 15 workstations and one Admin computer.

• Student laptops are connected to Admin laptop via a wireless router.

16

How is the WCBA Administered?

During the assessment….Students enter 10-digit ID to log in to the system.

NAEP staff monitors student’s progress on the Admin laptop.

17

WCBA Tutorial

18

• http://nces.ed.gov/nationsreportcard/writing/cba.asp

How is the WCBA Administered?

After the assessment….

• NAEP staff connects the Admin laptop to the Internet and transmits student data.

• NAEP staff connects the Admin laptop to the Internet and transmits student data.

19

How are students with disabilities and English-language learners assessed?

• Universal Design: features that are available to all students. These features are integrated into the system, and all students elect whether/when to use them.

• Accommodations: offered only to students who require them (SD or ELL). If provided through the computer, these features must be enabled.

20

How are students with disabilities and English-language learners assessed?Universal design elements on recent writing computer-based assessment include:

• Small group/one-on-one • Text to speech • Adjusting font size up to 48 points• Adjusting contrast/colors• Use of highlighter• Spell-check

21

How are SD and ELL students assessed?• Accommodation

elements on recent writing computer-based assessment include: • Extended time• Magnification greater

than 48 point font• Directions in Spanish• Breaks during test• Testing in a separate

session• Other

22

How was Universal Design received by schools and students?

• Many positive comments about being able to include students without singling them out.

• 20% fewer separate accommodation sessions in CBA then in paper-and-pencil assessments.

• Less burden on schools to find space and staff to administer accommodations.

• Students’ accommodations were less obvious to other students.

• Students were able to access accommodations as they chose.

23

How was the WCBA received by schools and students?• Students were

engaged in schools in varying demographics (setting, SES, etc.)

• Principals and teachers were excited to have their schools participate.

• Students used tools with ease.

24

Computer-Based Assessment of Technology and Engineering Literacy (TEL)

Malcolm BauerEducational Testing Service

Outline

• Technology and Engineering Literacy (TEL) Assessment

• Why Computer-Based Assessment (CBA)?• Innovations that CBA supports

• Reporting• Construct definition• Scoring• Task Development

26

TEL Assessment

“the capacity to use, understand, and evaluate technology …”

27

2010 Student/evidence model refinement

2011 Task model and task development

2012 Series of tryouts/pilots for task revision

2013 Large-scale psychometric pilot test

2014 Probe assessment

TEL Milestones

28

Innovation in Reporting

• Provide detailed, in-depth reporting of student competencies

• Provide information about cognitive processes and strategies

• Make results more “actionable” – better able to inform policy and practice

• Report these enhanced results in ways that are more informative and useful

29

Innovation in Process: Evidence-Centered Design

Student Model

What you want to measure

How to recognize and interpret observable evidence

How to elicit good evidence of what you want to measure

Evidence Model

Tasks

Can students design or redesign a device or system to address

a need?

Can students design or redesign a device or system to address

a need?

Designs are evidence of students’ ability.

Look to see whether students satisfied

requirements.

Designs are evidence of students’ ability.

Look to see whether students satisfied

requirements.

Have students design something.

Can they identify and meet the needs

(requirements)?

Have students design something.

Can they identify and meet the needs

(requirements)?

30

Innovation in Scoring

• Evidence of new competencies• Model-driven selected response• Evidence-rich work products

• Evidence of cognitive processes• Same as above• Pattern tracking

31

Item Types: Scenario-based Task and Discrete Items

TEL Domain

Long TaskLong Task

Long TaskLong Task

Long TaskLong Task

Long TaskLong Task

Short TaskShort Task

Short TaskShort Task

Short TaskShort Task

Short TaskShort Task Short

TaskShort Task

Short TaskShort Task

DD

DD

DD

DD DDDD

DD DDCRCR

DD

DD

DD

DD

CRCR

DD

DD

DD

CRCR

DD DD

DDDD

CRCR

DD

DD

DD

32

Innovation in Task Development

Scenario-based tasks• Are extended assessment activities

which combine multiple measures of student ability

• Leverage computer delivery to offer innovative, interactive assessment tools

33

Summary: Uses of CBA in TEL Assessment

• Allow students to perform educationally relevant and engaging activities

• Collect a variety of evidence of TEL competency and process

• Score student responses to tasks and interactions with tasks

• Provide access to TEL results that are informative and useful

34

Acknowledgements

• National Center for Education Statistics• National Assessment Governing Board• All members of the TEL ECD Team, the

TEL Task Development Team, and Measurement Team

• Hager Sharp Inc.

35

Questions?