Designing Math and ELA Tasks for AA-AAS Using ECD and UDL

This material is based on work supported by two Enhanced Assessment Grants from the U.S. Department of Education: Alternate Assessment Design—Mathematics and English Language Arts. The PADI online system was developed through a grant from the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the U.S. Department of Education or the National Science Foundation.

Designing Math and ELA Tasks for AA-AAS Using ECD and UDL

Designing Math and ELA Tasks for AA-AAS Using ECD and UDLRenée Cameto and Geneva Haertel – SRI InternationalPatricia Almond – University of Oregon

Wendy Carver – Utah State Office of EducationKaren Denbroeder – Florida Department of EducationCarol Scholtz – Idaho State Department of EducationDeborah Matthews – Kansas State Department of Education

Bob Dolan– Discussant

Renée Cameto and Geneva Haertel – SRI InternationalPatricia Almond – University of Oregon

Wendy Carver – Utah State Office of EducationKaren Denbroeder – Florida Department of EducationCarol Scholtz – Idaho State Department of EducationDeborah Matthews – Kansas State Department of Education

Bob Dolan– Discussant

National Conference on Student Assessment June 27, 2012

IntroductionIntroduction

Based on work from two Enhanced Assessment Grants funded by the U.S. Department of Education

Each project involved a consortium of states and SRI International: Alternate Assessment Design–Mathematics

Utah, Idaho, Florida

Alternate Assessment Design–English Language Arts Idaho, Utah, Kansas

2

AgendaAgenda ECD/UDL Frameworks Co-Design Process Example Design Patterns and Item Suites

Math English language arts (ELA)

Item Suite Tryouts Teacher and student background characteristics Student performance Teacher feedback on items

Discussant Q & A

3

NeedNeed

Alternate Assessments need to be well designed The design and development processes

employed need to be systematic, rigorous, and reflective of industry standards

NCLB/ESEA require that students with disabilities, including SWSCD, be assessed on grade-level academic content

Assessment continues to be a lever of educational change (assessment drives instruction and school reform)

4

Innovative ApproachInnovative Approach Evidence-centered design (ECD) is an

innovative assessment design process Initial work on ECD conducted at ETS by Mislevy,

Steinberg, and Almond (2003) Has been used for more than 15 years; in these EAG

projects, ECD is being extended to the population of students with significant cognitive disabilities

Is a framework and set of processes Meets criteria for developing well-designed

assessments Supports the design of items that are aligned to the

focal constructs of interest

5

Innovative ApproachInnovative Approach ECD can be applied to:

All subject areas All grade levels All types of assessments (large scale, summative,

formative, technology-enabled, pencil/paper, advanced placement, workforce, etc.)

All types of alternate assessments (portfolio, performance task, checklist, etc.)

All item/task formats (multiple choice, constructed response, performance task, etc.)

Integrating UDL into the ECD framework promotes accessibility of items through consideration of student needs and abilities during initial design and throughout the design process

6

What is Evidence-Centered Design?What is Evidence-Centered Design?

Critical Question: How do we judge what students know and what they can do? Make explicit what knowledge or skills are the target Identify non-target but required knowledge and skills Generate the kinds of observations that provide

evidence of a student having the target knowledge Determine the kinds of stimuli and work products

needed to gather the evidence

These attributes form the basis of the assessment argument - an argument based on evidence

7

Co-Design ProcessCo-Design Process ECD process makes

use of co-design at all layers

Co-design team typically includes: Special educators

(experience instructing SWSCD in academics)

Content experts ECD experts State assessment

administrators

8

Alternate Assessment DesignAlternate Assessment Design

A multistep process using ECD co-design

9

Select Standards

Common Core

Domain Analysis

Create Design Patterns

Domain Modeling

Create Summary Task

TemplatesConceptu

al Assessme

nt Framewo

rk

Author Assessment

Tasks

Assessment Implementati

on

Pilot and Refine Tasks

Assessment Delivery

Select StandardsSelect Standards

GOAL: In Domain Analysis, content relevant to the assessment is organized and selectedIdentify standards to meet AA-AAS needs

AAD-Math: Used NCTM Standards and Expectations later created crosswalk to Common Core State Standards

AAD-ELA: Common Core State Standards and the North Carolina Common Core State Standards

10

Select Standards

Common Core

Create Design PatternsCreate Design Patterns

GOAL: In Domain Modeling, a narrative description of the assessment argument structure (Design Pattern) is produced to guide task developmentDesign Patterns for selected standards are developedDesign Patterns are reusable and improve efficiency of task developmentDesign Patterns can improve content validityDesign Patterns create a design space for assessment developers

11

Create Design

Patterns

PADI Online Design SystemPADI Online Design System

12

Elements of Design PatternsElements of Design Patterns

Student Model What knowledge, skills or abilities

(KSAs) should be assessed?

What construct (complex of student attributes) should be assessed?

Focal KSAs

Primary Knowledge, Skills, and Abilities (KSAs) targeted by this DP

Additional KSAs

Other KSAs that may be required by tasks from this DP, some of which can be supported by UDL and accommodations

Educational Standards

Associations with Educational Standards from different states as well as national standards, if desired

13


Student Model What knowledge, skills or abilities

(KSAs) should be assessed?


Focal KSAs Primary Knowledge, Skills, and Abilities (KSAs) targeted by this DP

Additional KSAs

Other KSAs that may be required by tasks from this DP, some of which can be supported by UDL and accommodations

Educational Standards

Associations with Educational Standards from different states as well as national standards, if desired

14


Evidence Model What behaviors or performances should

reveal the knowledge, skills, and abilities?

What behaviors should reveal the construct?

Potential Observations

Observed behaviors of students that can provide evidence of Focal KSAs

Potential Work Products

What students say, do, or make that provides evidence about the Focal KSAs

Potential Rubrics Some evaluation techniques that may apply

15


Evidence Model What behaviors or performances should

reveal the knowledge, skills, and abilities?

What behaviors should reveal the construct?

Potential Observations

Observed behaviors of students that can provide evidence of Focal KSAs

Potential Work Products

What students say, do, or make that provides evidence about the Focal KSAs

Potential Rubrics

Some evaluation techniques that may apply

16


Task Model What tasks, situations, or stimuli should

elicit those behaviors and performances?

What tasks should elicit those behaviors?

Characteristic Features

Aspects of assessment situations likely to evoke the desired evidence

Variable Features Aspects of assessment situations that can be varied in order to control difficulty or target emphasis on various aspects of KSAs

17


Task Model What tasks, situations, or stimuli should

elicit those behaviors and performances?

What tasks should elicit those behaviors?

Characteristic Features

Aspects of assessment situations likely to evoke the desired evidence

Variable Features

Aspects of assessment situations that can be varied in order to control difficulty, target emphasis, or provide support for Additional KSAs

18

Relationship of Design Pattern ElementsRelationship of Design Pattern Elements

Variable Features are the aspects of the task that are varied to support the Additional KSAsUDL Category

Example Additional KSA

Example Variable Feature

Perceptual (Receptive)

Ability to perceive linguistic components of the stimulus material and question

Delivery mechanisms by which question is perceived (e.g., read aloud, concrete objects, Braille)

Skill and Fluency (Expressive)

Ability to communicate response (e.g., respond verbally, by using pictures, by making a selection from a group)

Response mode options (e.g., pointing, speech and verbalization, assistive device/augmentative communication)

Language and Symbols

Ability to decode text, symbols, or images

Level of abstraction required of student (e.g., concrete objects, images, text)

19

Relationship of Design Pattern ElementsRelationship of Design Pattern Elements

Variable Features are the aspects of the task that are varied to support the Additional KSAs

UDL Category Example Additional KSA

Example Variable Feature

Cognitive Ability to process multistep problems

Provide graphic organizers

Executive Ability to plan and sequence

Prompts and scaffolds to estimate effort, resources, and difficulty

Affective Ability to engage (e.g., task-specific motivation)

Enhance relevance, value, and authenticity of tasks

20

Source: Center for Applied Special Technology (CAST), http://CAST.org

Author Assessment TasksAuthor Assessment Tasks

GOAL: In Assessment Implementation, tasks are authoredInformation pre-populates from the Design Pattern to the Task TemplateTemplates are used to author a family of 4 items aligned with Focal KSA

Complexity is graduated within the family of four items

UDL is incorporatedItems individually administered

21

Author Assessment

Tasks

Author Assessment TasksAuthor Assessment Tasks Step 1. Pre-populate Task Template with

information from the Design Pattern Focal KSAs Additional KSAs Potential Observations and Work Product Variable Features for Cognitive Background and UDL Characteristic Features

Step 2. Select attributes for authoring items Focal KSA Additional KSAs Potential Observations and Work Products for each item Variable Features for Cognitive Background and UDL

22


Step 3. Select Variable Features to be used in the item family Consider ways to vary complexity

(e.g., limit numbers to those with two or fewer digits, limit number of words in a passage

Consider ways to incorporate UDL (e.g., include multiple representations of stimulus materials)

23


Step 4. Create Item

24

Mathematics Grade 9-12 RecyclingMathematics Grade 9-12 Recycling

25


Focal KSAs • Ability to identify and/or calculate summary statistic to answer a question when given a data set and a question about the data

• Ability to answer a question about data by identifying, creating, and using a graphical display, and calculating the summary statistic

Additional KSAs

• Knowledge of what bar graphs, histograms, dot plots, stem-and-leaf and box plots are

• Ability to create graphical representations (e.g., bar graphs, histograms, dot plots, stem-and-leaf and box plots) of a set of data

• Ability to use (read and interpret) graphical representations (e.g., bar graphs, histograms, dot plots, stem-and-leaf and box plots) to answer questions about the data

• Ability to select representations (e.g., bar graphs, histograms, dot plots, stem-and-leaf and box plots) of different types of data (e.g., frequency data, univariate data) to answer specific questions about the data

• Knowledge of prerequisite vocabulary and symbols, and basic understand of concept (e.g., median, outliers, spread, minimum and maximum, range, box plot, stem and leaf, mode)

• Knowledge of what data are (e.g. a number that represents a property of some item)

• Ability to add, subtract, multiply and divide

CCSS

NCTM

• For univariate measurement data, be able to display the distribution, describe its shape, and select and calculate summary statistics

• For univariate measurement data, be able to display the distribution, describe its shape, and select and calculate summary statistics.

Mathematics Item C – part 1Mathematics Item C – part 1

26


27


28

Mathematics Item BMathematics Item B

29

Mathematics Item A1Mathematics Item A1

30

Mathematics Item A1Mathematics Item A1

31

Item Directive Stimulus Materials

If student cannot respond, the examiner says, “Look at/touch the bar graph.”


Step 5. Document correct answer

Step 6. Describe stimulus materials

Step 7. Describe materials for the examiner

32

ELA Reading Grade 3 Charlotte’s WebELA Reading Grade 3 Charlotte’s Web

33


Focal KSAs Ability to describe the characteristics of the characters in a story, with an emphasis on characteristics such as feelings, traits, or motivations.

Additional KSAs

AK1. Ability to identify characters in a storyAK2. Knowledge that a story has characters with traits (e.g., age, gender, height, nice, mean), motivations (e.g., intangible reasons for taking an action or doing something), and feelings (e.g., sadness, happiness)AK3. Knowledge of characteristics, such as traits, motivations, or feelings

CCSS

NCECCSS

Describe characters in a story (e.g., their traits, motivations, or feelings) and explain how they contribute to the sequence of events.

Identify the feelings of characters in the story.

ELA Item C- part 1ELA Item C- part 1

34

Item Directive Stimulus MaterialsExaminer presents student with a printed passage adapted from Charlotte's Web and says, We are going to read a passage from a book called Charlotte's Web. Listen for the feelings of the main character, Wilbur. Examiner reads the following aloud: Wilbur, the pig, lived in a big barn. He lived with horses, geese, sheep, and a rat. Even so, Wilbur was lonely. He started to cry. Then all of a sudden, a voice said "I will be your friend." She said her name was Charlotte. Wilbur liked having a new friend.Examiner places the passage where the student can see it.

a) Examiner presents student with three note cards and says, How did Wilbur feel before he met Charlotte? Examiner points to each option and reads it aloud:- tired - sad - excited

Wilbur, the pig, lived in a big barn. He lived with horses, geese, sheep, and a rat. Even so, Wilbur was lonely. He started to cry. Then all of a sudden, a voice said, "I will be your friend." She said her name was Charlotte. Wilbur liked having a new friend.

tired sad excited

ELA Item C- part 2ELA Item C- part 2

35

Item Directive Stimulus MaterialsExaminer presents student with the same printed passage adapted from Charlotte's Web and says, We are going to read a passage from a book called Charlotte's Web. Listen for the feelings of the main character, Wilbur. Examiner reads the passage aloud: Wilbur, the pig, lived in a big barn. He lived with horses, geese, sheep, and a rat. Even so, Wilbur was lonely. He started to cry. Then all of a sudden, a voice said "I will be your friend." She said her name was Charlotte. Wilbur liked having a new friend.Examiner places the passage where the student can see it.

b) Examiner presents student with three note cards and says, How did Wilbur feel after he met Charlotte? Examiner points to each option and reads it aloud:- happy - sad - scared

Wilbur, the pig, lived in a big barn. He lived with horses, geese, sheep, and a rat. Even so, Wilbur was lonely. He started to cry. Then all of a sudden, a voice said, "I will be your friend." She said her name was Charlotte. Wilbur liked having a new friend.

happy sad scared

ELA Item BELA Item B

36

Item Directive Stimulus MaterialsExaminer presents student with a printed passage adapted from Charlotte's Web and says, We are going to read a passage from a book called Charlotte's Web. Listen for what Wilbur, a character in the book, wishes he could do. Examiner reads the passage aloud: Wilbur watched Charlotte, the spider, spin her web. Wilbur wished he could make a web like Charlotte. He could not because he was a pig, not a spider. Examiner places the passage where the student can see it.Examiner presents student with three note cards and says, What did Wilbur wish he could do? Examiner points to each option and reads aloud:- Spin a web- Eat food- Crawl like a spider

Wilbur watched Charlotte, the spider, spin her web. Wilbur wished he could make a web like Charlotte. He could not because he was a pig, not a spider.

Spin a web

Eat food

Crawl like a spider

ELA Item A1ELA Item A1

37

Item Directive Stimulus MaterialsExaminer presents student with a printed passage adapted from Charlotte's Web and says, We are going to read a passage from a book called Charlotte's Web. Examiner reads the passage aloud: Wilbur, the pig, lived in a big barn. Examiner places the passage where the student can see it. Examiner presents student with two picture cards (pig, barn) and says, This is a picture of a pig. This is a picture of a barn. [Show me] / [Touch] / [Look at] the picture of the character from Charlotte’s Web.

Wilbur, the pig, lived in a big barn

ELA Item A2ELA Item A2

38

Item Directive Stimulus Materials

If student does not respond to A1, Examiner removes the picture of the barn, points to the passage and re-reads the passage aloud: Wilbur, the pig, lived in a big barn. Examiner says, [Show me] / [Touch] / [Look at] the picture with the character from the story.

Wilbur, the pig, lived in a big barn

Library of Design Patterns and Items SuitesLibrary of Design Patterns and Items Suites AAD-Math and AAD-ELA projects produced:

50 Design Patterns 50 Item Suites – 200 items

Spans grades 3–8 and high school Math Design Patterns organized around NCTM

Standards and Expectations cross-walked to CCSS

ELA Design Patterns organized around Common Core State Standards

Using multiple-choice, open-ended and scenario-based formats

39

Benefits of Using ECD with UDLBenefits of Using ECD with UDL

What is gain from applying this approach in assessment design: Content-valid assessment tasks designed to

align to state or national standards Increased efficiency through systematic task

design and development-reusable design patterns and task templates

Built-in documentation of design decisions using PADI (NSF and SRI development funding)

Variation in complexity and DOK Attention to UDL throughout design process

40

Pilot and Refine TasksPilot and Refine Tasks

GOAL: In Assessment Delivery, the newly developed assessment tasks must be pilot tested and refined based on empirical resultsAll newly developed assessment tasks must be empirically studied to establish their feasibility, reliability, and validityMethodologies may include cognitive labs, teacher surveys, field tests of tasks, and student observations

41

Pilot and Refine Tasks

Item Suite Tryouts Research Questions Item Suite Tryouts Research Questions

Task and Item Viability Can the suite of items be administered as

designed? Are the items clear to the student and to the

teacher? Appropriateness for a Range of Student

Performance Levels Can any of the students do the most difficult

item? Can most students do the least difficult item?

42

Item Suite TryoutsItem Suite Tryouts Detailed Materials and Instructions for teachers Teacher completes Data Collection Booklet:

Performance for each student on each item administered

Item related information Student received instruction related to the item Item viability–appropriateness of directions,

graphics, manipulatives and materials, language, complexity

Student background information Communication level, disability, grade-level

Teacher background information Years of experience, familiarity with academic

standards43

Flow ChartFlow Chart

There are two paths:

Student responds correctly to A1

Student responds incorrectly to A1

Administer

Item A1

Administer

Item A2

Stop administration

Stop administration

Correct response

Administer

Item B

Administer

Item C

Stop administration

Correct, incorrect, no response, or refused

Incorrect or no response

Correct, incorrect, no response, or refused

Note: Be sure to gain the student’s attention before presenting EACH item.

Refused

44

How to Administer Suite of ItemsStart here

Sequence of Items within a Suite When Item A1 Is Answered Correctly

Sequence of Items within a Suite When Item A1 Is Answered Correctly

45

Sequence of Items within a Suite When Item A1 Is Answered Incorrectly

Sequence of Items within a Suite When Item A1 Is Answered Incorrectly

46

School Types and Teachers DegreesSchool Types and Teachers Degrees

School TypePercen

tElementary school 50%Middle school/junior high 32%High school 27%Special school 4%

47

Teachers DegreesPercen

tMaster's 60%Bachelor's 36%Advanced degree beyond master's

4%

Percent of Teachers with 5 or More Years Experience Percent of Teachers with 5 or More Years Experience

Teaching ExperiencePerce

ntTeaching 86%Teaching students with significant cognitive disabilities

77%

Teaching Functional English language arts to students with significant cognitive disabilities

70%

Teaching Grade Level English language arts to students with significant cognitive disabilities

51%

48

Item Suite Tryout Study SampleItem Suite Tryout Study Sample

Teachers StudentsState Number Percent Number Percent

State 1 35 37% 72 23%

State 2 21 22% 109 35%

State 3 40 42% 127 41%

Total 96 308

49

Students to Teacher Ratio by State

State 1 ~ 2:1 State 2 ~ 5:1 State 3 ~ 3:1

Percent of Students at Three Levels of Communication

Percent of Students at Three Levels of Communication

Level Definition Number Percent

Low Awareness: Has no clear response and no objective in communication; Pre-symbolic: Communicates with gestures, purposeful moving to object, sounds

65 21%

Middle Concrete Symbolic: Beginning to use pictures or other symbols to communicate within a limited vocabulary

76 25%

High Abstract Symbolic: Speaks or has vocabulary of signs, pictures to communicate. Recognizes some sight words, numbers, etc.

167 54%

Total 308 100%

50

Percent of Students in Type

of Educational Setting

Percent of Students in Type

of Educational Setting

51

Educational Setting PercentSelf-contained class in regular school

77%

Special school 10%

Resource room 8%

Inclusive setting 4%

Total100%

Percent of Students in Grade SpansPercent of Students in Grade Spans

52

n = 308

Students’ Primary DisabilityStudents’ Primary Disability

Primary DisabilityPerce

ntIntellectual disability 45%Autism 21%Multiple disabilities 21%Other health impairment

5%

Specific learning disability

4%

Low Incidence Disabilities

5%

100%53

Number of Item Suites Administered by Grade Span and Strand

Number of Item Suites Administered by Grade Span and Strand

54

CCSS Strands ElemMiddl

e High

Total

Reading 153 90 89 332Language 79 74 42 195Speaking/Listening

30 31 47 108

Writing 91 39 130 353 195 217 765NOTE: 308 students took 765 tasks, i.e. many students took more than one task 96% implementation fidelity

Reading Item Suite AdministrationsReading Item Suite Administrations

55

Language Item Suite Administrations Language Item Suite Administrations

56

Speaking & Listening Item Suite Administrations

Speaking & Listening Item Suite Administrations

57

Writing Item Suite AdministrationsWriting Item Suite Administrations

58

Student Path by Communication LevelStudent Path by Communication Level

Communication LevelPath High Medium Low Overall

A1 & A2

29% 33% 39% 43%

A, B, & C

71% 19% 10% 55%

Total 52% 25% 23%

59

Which path did students take?

ELA Item Suite TryoutsELA Item Suite Tryouts

60

Percent of Items Correct by Student Communication Level

Did student receive instruction [opportunity to learn] in ELA content and skills in this [item]area?

Did student receive instruction [opportunity to learn] in ELA content and skills in this [item]area?

Yes No DK/NR OverallA1 309 391 65 765 40% 51% 8% B 206 169 42 417

49% 41% 10% C 186 185 41 412

45% 45% 10%

61

DK = Don’t Know, NR = No Response

Teachers’ Feedback by Item

A1 B C (n ~ 750) (n ~ 405) (n ~ 400)

Teacher Survey Question % % %• Item was interesting and

engaging for this student68% 79% 72%

• Student's response to item was clear and observable

95% 96% 96%

• Item was appropriate for this student with significant cognitive disabilities

64% 69% 64%

62

Teachers’ Feedback by ItemItem Suitability

Assessment Directions and Materials Assessment Directions and Materials

A1 B C (n ~ 750) (n ~ 405) (n ~ 400)

Teacher Survey Question % % %• Test directions for teachers

provided just the right amount of info

88% 87% 87%

• Stimulus materials supported the student's understanding of item

72% 73% 70%

• Size of the stimulus materials was just right

93% 93% 91%

• Amount of detail in stimulus materials was just right

82% 79% 72%63


Item Context and ComplexityItem Context and Complexity

64


A1 B C (n ~ 750)

(n ~ 405) (n ~ 400)

Teacher Survey Question % % %• Item context was understandable to the

student 62% 71% 63%

• Item context was helpful to the student 59% 70% 63%• Item context was appropriate for the

student 65% 73% 68%

• Language used in the item was just right

65% 71% 66%

• Amount of effort required of the student in the item was just right

76% 78% 71%

• Number of steps in the item just right 78% 80% 73%• Content knowledge required by the

item was just right

60% 68% 61%

SummarySummary ECD is well-suited to developing alternate

assessment tasks; supports integration of UDL ECD allows the systematic documentation of

assessment tasks to support efficiency of task development (re-usability)

ECD supports the design/development of a range of items with varying levels of complexity and DOK

The co-design approach actualizes the value of the special educators, content specialists and assessment specialists

65

Contact InformationContact Information Renee Cameto, Ph.D.

[email protected] http://alternateassessmentdesign.sri.com/

66

Designing Math and ELA Tasks for AA-AASUsing ECD and UDL

Discussion

Designing Math and ELA Tasks for AA-AASUsing ECD and UDL

Discussion

CCSSO NCSA 2012

Bob DolanPearson

Slide 68

What is Universal Design?What is Universal Design?

Ron Mace (1991) North Carolina State University.

Beginnings in field of architecture. Design philosophy: consider variety of ways

in which individuals will use navigate in physical spaces and use physical devices.

One size doesn’t fit all. Retrofits are expensive and limited. Benefits all users, not only subgroups.

Slide 69

Universal Design for LearningUniversal Design for Learning

Extends concept of universal design from a physical space to a pedagogical space.

Relies on flexible scaffolds and accommodations to support students’ challenges and minimize construct-irrelevant barriers.

Supports individual learner differences by providing: Multiple means of recognition. Multiple means of interaction & expression. Multiple means of engagement.

It’s not just about special populations (i.e., English learners and students with disabilities). However, it can guide how to best consider these students

from a capabilities/challenges perspective rather than an “label” perspective.

Slide 70

UDL and AssessmentUDL and Assessment How do we provide multiple means of recognition, interaction &

expression, and engagement? By considering the following categories of student processing and

interaction: Perception — Conversion and categorization of sensations into

stable and valid representations. Linguistic Skills — Recognition of the patterns of auditory, visual,

and tactile stimuli that constitute language. Cognition — Construction of meaning from the elements of

perception and language. Motor Skills — Construction of meaningful patterns of action. Executive Processing — Goal setting & maintenance, planning &

strategizing, effort & mental resource allocation & organization, and progress monitoring.

Affect — Evaluation of the importance or significance of events, objects, or plans.

Slide 71

UDL and ECDUDL and ECD

Chocolate & peanut butter UDL provides an explicit framework for

minimizing the introduction of construct-irrelevant variance into student scores by reducing or eliminating the impact of additional KSAs on observations and work products.

This has also been done for general assessment (IES grant to Haertel, Mislevy, Dolan)

Complexity of ECDComplexity of ECD

Perhaps initially overwhelming But logical and reasonable, and

necessary and sufficient for validity. Puts hard work up front.

PADI system scaffolds the process for item developers. But still proprietary?

Slide 72

Walking the AA-AAS TightropeWalking the AA-AAS Tightrope

Maintain grade-level content alignment while identifying and reducing impact of construct-irrelevant variance.

ECD: Explicating the additional KSAs provides balance.

UDL: Considering student interaction and processing categories provides traction.

Slide 73

Co-Design ProcessCo-Design Process

Team approach critical to success (special educators, content experts, ECD experts, state assessment administrators).

Both ECD and UD approaches require concurrent application of expertise.

Slide 74

ApplicabilityApplicability

Supports development of performance task and portfolio assessments.

Slide 75

MethodologyMethodology

Diverse student sampling (primary disability, classroom type, teacher experience, …)

Initial qualitative/quantitative evaluation approach appropriate. Maximizes opportunities for informed

revision. Branching administration approach

maximizes opportunity to learn about item performance across diverse students.

Slide 76

ResultsResults

Suggest quantitative correlations between students’ communication level (based on teacher evaluation) and “path.” But would need finer “path” granularity? And would it be overkill for item tryout?

(Though it might provide additional information for item revision.)

Teachers’ feedback Expected stronger correlations between

student “paths” and Item Context and Complexity criteria.

Slide 77

Next StepsNext Steps Field testing.

Is it any different than for other AA-AASs? Can item tryout methodologies be applied?

Scaling up. How well can this be done by ECD outsiders? Is this applicable to checklist AAs?

Efficacy studies How do we show that this approach is better/best?

“Learning from the margins” (Meyer & Rose, 2005): taking what we’ve learned from considering special populations and applying it to the general population.

Slide 78

Documents

Designing Math and ELA Tasks for AA-AAS Using ECD and UDL