California English Language Development Test: Developing the Early Literacy Assessment Components

CALIFORNIA DEPARTMENT OF EDUCATIONJack O’Connell, State Superintendent of Public Instruction

California English Language Development Test:

Developing the Early Literacy Assessment Components

Panel Presentation at the Tenth Annual Accountability Leadership Institute

Burlingame, CADecember 8, 2009

JACK O’CONNELLState Superintendent of Public Instruction

K-1 Early Literacy Assessment 2

Section 1 – Introductions, Background, and Policy

Requirements

Lily Roberts, Administrator,California English

Language Development Test (CELDT) Office



Presentation Overview

• Background and policy requirements• Item development cycle• Guiding principles for test

development and administration• Field test and lessons learned• Development and technical issues• Discussion• Questions and answers



Background• 2007: State legislature authorized

development of the Kindergarten and grade one (K-1) early literacy assessment

• 2008: Initial item development and expert panel was convened

• 2009: Field test was conducted in March

• 2009-10: Operational testing in all four domains for all grades (K-12)

• 2010: Standard setting will be conducted to recommend performance levels



Policy Requirements

• Title III, Elementary and Secondary Education Act requires testing in the domains of listening, speaking, reading, and writing in all grades

• California Education Code (EC) Section 60810 defines the purposes for the CELDT:– Identify limited English proficient students – Determine the level of English language

proficiency– Assess progress acquiring the skills of

listening, speaking, reading, and writing



Reclassification Criteria

• EC Section 313 specifies four criteria for local reclassification decisions, including:– Assessment of English language

proficiency– Performance in basic skills– Teacher evaluation– Parental opinion and consultation



Section 2 - Initial Item Development

Edynn Sato, WestEd



2008 Item Development

• Subject matter experts participated in guiding item development

• Conducted two rounds of content and bias reviews

• Developed 142 K-1 reading and writing test questions

• Twelve reading and writing English Language Development (ELD) standards were identified as important and assessable for K-1 by subject matter experts



Item Development Cycle

• Test specifications are created based on ELD standards

• Guiding principles for item development are established

• Item specifications are created to guide item development

• Item writers and editors receive training

• Items are written and edited

• Graphics are created



Item Development Cycle (cont.)

• Lead editors and content director review and sign-off on items that are ready for CELDT Office and committee reviews

• Interdepartmental staff reviews items/graphics

• Subject matter experts review items for content

• Subject matter experts review items for bias

• Necessary edits are made



Guidelines for Item Writing• General Guidelines

– Typical representations– Reflect a range of cultural and ethnic

diversity– Familiar to a range of students

• Specific Guidelines– Specific, detailed, and clear– Contextual detail to enhance

comprehensibility– People in graphics

• Age, gender, ethnicity



Specific Questions that Need to be Asked during Content Review

• Is the item aligned to the correct ELD Standard?

• Does the item test a knowledge or skill that is appropriate for the assigned domain?

• Is the item aligned to the correct grade span?

• Does the item test for current and worthwhile concepts?

• Does the item stem give a full sense of what is being asked?



• Are the directions clear?• Does the stem avoid wordiness? • Is the key the only correct answer?• Does the item use distractors that relate

to the stem in the same way as the key?

• Does the item use distractors that are plausible misconceptions and errors?

• Does the item contain clues to the correct answer?

Specific Questions (cont.)



Specific Questions (cont.)

IF THE ITEM CONTAINS GRAPHICS…

• Is the graphic required to answer the item?

• Does the graphic provide sufficient information needed to answer the item?



Item Bias and Sensitivity Review

• Ensure that the items avoid stereotype patterns

• Ensure that the items avoid language, symbols, words, phrases, or examples that are biased or potentially offensive, inappropriate, or negative toward any group

• Ensure that the items acknowledge the multicultural nature of our society

• Ensure items do not unfairly advantage or disadvantage any group



Different Types of Bias

• Bias can refer to age, gender, race, ethnicity, cultural background, social-economic background, religion, disability, or mental condition.

• Bias in test items can be present as Content bias or Language bias.



Additional development activities:

• Cognitive interviews of students• Interviews of teachers • Small-scale pilot of items



Section 3 - Field Test and Lessons Learned for

Future CELDT Editions

Caroline Fahmy,Educational Data Systems,

CELDT Test Contractor



Overview: Lessons Learned K-1 Early Literacy

• Review of test and lessons learned related to:– Spring 2009 field test– 2009-10 Operational test– Modifications to be incorporated into

the 2010-11 operational test



2009 Field Test (FT)

• 142 items were delivered to the new CELDT contractor in February

• Conducted FT March 16-20, 2009

• Constructed four FT forms with ten reading and ten writing items each

• Administered FT in 37 districts to 2,548 K-1 students in 57 schools across California



Number of Field Test Participants by Grade

FORM Kinder-garten

Grade One Total

1 274 362 636

2 370 296 666

3 287 322 609

4 347 290 637

Total 1,278 1,270 2,548



2009-10 Lessons Learned: Test Materials

• Format student book to eliminate distractions and enhance ease of administration

• Provide all specific, test question information in the student book for ease and speed of administration



1Examiner Page

Student Page

2 1

2

Page Layout Changes



2009-10 Lessons Learned:Test Examiner

• Include examples on how to pronounce the letter sounds next to each item script

• Allow examiner to repeat any question, but no prompting allowed

• Provide allowable prompts in the sample practice items in the student test book



2009-10 Lessons Learned:Student

• Use formatting to minimize distractions

• Use sample item administration as a time to really practice the items

• Do not underestimate students’ English language ability



2010-11 Lessons Learned: Test Materials

• Student Book and Examiner’s Manuals will include all four domains

• Examiner’s Manual will contain all instructions, questions, and pictures

• Student Book will only contain:– Bubbles for responses– Areas for written responses



2010-11 Lessons Learned: Test Materials (cont.)

• Student Books will be greatly reduced in size

• Scoring Guide will be combined with Examiner’s Manual (for all grades)



2010-11 Lessons Learned: Test Examiner

• Move instructions from Student Book to Examiner’s Manual

• Refine the “Acceptable Responses Include” section (based on actual student responses)

• Review the use of pictures in the writing domain



2010-11 Lessons Learned: Student

• Writing area - modify open box design to:– Help student know where to write– Help administrator and scorer better

interpret upper case and lower case letters– Keep all writing in area clipped for image

scoring

• Use page formatting to accommodate left-handed students in copying words



Section 4 - Technical Considerationsand Discussion

Richard Duran, University of California, Santa Barbara

Member, CELDT Technical Advisory Group



K-1 Expert Panel and Technical Advisory Group

• Input on development issues– Guiding principles– Alignment to ELD Standards

• Input on technical issues– Pilot test– Field test– Operational test



Primary Development Issues

• Guiding Principles and ELD Standards:

Items must:– embody ELD Standards– be based on early literacy theory/research– be meaningful to EL K-1 students– easily administered– scored reliably

• Other Concerns



Technical Issues• December 2008 pilot test on small K-1 sample of

English learner and English-only students– p-values computed– Cognitive interviews conducted– Screened out some items and revised others– Procedures for administration reviewed

• March 2009 K-1 field test of four forms

- Ten reading and ten writing items per form- Reading items had already been calibrated on

common reading scale- Computed basic test statistics, p-values, point

biserial correlations



Technical Issues (cont.)

• July 2009 implemented operational test and a linking study for K-1 writing items with a sample of grade two English learners

• January 2010 will conduct a standard setting for K-1 reading and writing performance-level cut scores using a modified Bookmark process



Discussion

• Achieved alignment of K-1 assessments with state and federal policy and professional testing standards

• Contributed to state and national efforts to better ground English language development assessment and practice



For More Information

Lily Roberts, Ph.D., Administrator

CELDT Office

Statewide Assessment Division

Telephone: 916-319-0784

E-mail: [email protected]

Web: www.cde.ca.gov/ta/tg/el

Documents

California English Language Development Test: Developing the Early Literacy Assessment Components