1

Assessment Directors MeetingAugust 10, 2006

Judy W. ParkAssessment & Accountability Director

Utah State Office of EducationJuly 2006

2

Welcome

Introductions

District Sharing Cards

3

District

Assessment Handbook

4

Put Students in the Drivers Seat

• Clear Core Curriculum

• Clear Test Purpose

• Quality Assessments• Error Free• Available on time

• Trained Administrators

5

Put Students in the Drivers Seat

• Testing Environment• Standard Administration• All tools available

• Immediate Results

• Quality Results Interpretation

• Test Results are clearly connected to preparing for the next test

6

Put Students in the Drivers Seat

• Clear Core Curriculum

• Clear Test Purpose

• Quality Assessments

• Trained Administrators

• Testing Environment

• Immediate Results

• Quality Results Interpretation

• Test Results are clearly connected to preparing for the next test

7

Fall Road Trip • District Responsibility

• Invite A & A• Host meeting – computer lab setting

• Audience – Determined by District• School Administrators?• School Test Coordinators?• School Counselors?• Teachers?

• Agenda• Put Students in the Driver’s Seat• Assessment & Accountability Literacy• Standard Test Administration & Test Ethics• Understanding Summative Test Reports• Using U-PASS & other data to inform instruction

& improve student achievement

8

Test Development• August – Blueprint

• September - Item Development

• October – Item Review (Content)

• November – Bias & Sensitivity Review

• January – Construct Pilot Test Form Take Test

• February - Data Review

• March – Construct Operational Test Form Take Test

• May – Standard Setting, Scoring, Reporting Review Test Results

9

Test Development

August – Blueprint

Just as a blueprint of a house delineates the house’s framework, the test blueprint outlines the test’s framework.

10

Decision Guidelines for Blueprint Construction

• The standards and objectives selected for the blueprint must represent the entire curriculum.

• Only those objectives which can clearly be assessed in a multiple-choice format are selected for the blueprint.

• The number of test items assigned to each objective reflects the depth and breadth of the indicators within the objective.

• The following questions should be asked:

What proportion of the items on a test should represent each standard of the core curriculum?

What proportion of the items should represent each objective?

11

Decision Guidelines for Blueprint Construction

• The total number of questions on a test must be large enough to provide a fair sample of student performance across the standards and objectives.

• We need to determine the appropriateness of making inferences about a student’s proficiency with respect to the standard and objective based upon “x” number of items.

• Items are aligned to the standards and objectives and not the indicators.

12

Test Aligns to Standards and Objectives

• Item writers first refer to the core curriculum, specifically to the portions of the core assessed on a CRT.

• Standards are very broad, educational goals.• Objectives are broad statements of what a student

should be able to do.• Indicators are the descriptions of how students

demonstrate that they can perform the objective.• There are multiple indicators for each objective, which

provide a broad spectrum of content skills assessed on each year’s test.

• Indicators provide a reference for composing test items.

13

Caveats

• Every blueprint shows:• The Standard

• The Objective• The number of questions on the CRT for

each objective

• Only those objectives which can clearly be assessed in a multiple-choice format are selected for the blueprint.

• The number of test items assigned to each objective reflects the depth and breadth of the indicators within the objective.

14

Assessment Updates

• IOWA

• UBSCT

• Math CRT

• UTIPS

• NAEP

15

Accountability Update• Applied Math 1 & 2

• Not included in accountability calculations

• August 15?• Electronic Reports on ftp site• Research disk

• August 20 – 30• CRT Reports & Profiles electronic copy• Contact Sharon Marsh if you want hard copies

• September 20• Cumulative Student Report electronic copy• Contact Sharon Marsh if you want hard copies

16

Accountability Update

Accountability Meetings

for

U-PASS, AYP, AMAO

• Sevier District Office – August 18• Granite District Office – August 22• Weber District Office – August 23• Nebo Learning Center – August 25

17

U-PASS 2006 Reports

• May Board Meeting• Request to explore changes

• June Board Meeting• Changes approved

18

U-PASS 2006 Reports

• Participation 95%

• Status 80%

• Progress 190

• Confidence Interval 95%

Projection - Identify 12.7% of schools

19

U-PASS High School Task Force

June 5, 2006

Minutes from this meeting are in red

20

2006 Timeline• April 2006

• Meeting to review designs and make decisions

• May – July• Draft business rules

• August Assessment Directors Meeting• Share business rules

• August – October• USOE run all of the schools through the business rules• Districts may replicate analyses in their districts• USOE run multiple scenarios for status/progress

• November 14—Task Force Meeting!• Meeting to review and clean up the business rules• Review data and the multiple scenarios• Make initial decisions for status & progress

21

2007 Milestones• January - February 2007

• USOE Rerun all reports and fix all business rules• Districts replicate the rules for their districts

• March 2007—Task Force Meeting!• Meeting to review data and business rules• Final planning meeting to approve business rules and cutscores

• April 2007• Develop initial school designations• Meeting for districts to review all school designations• Make any final adjustments to business rules and calculations

• July 15• Run all AYP/U-PASS/AMAO Reports

• August 15 • Release all Accountability to districts for 30 day review

• September 15• Release Accountability reports to the public

22

U-PASS Purpose

• To identify schools in need of assistance to meet state standards

• This is a “minimum competency” goal. It will be important to keep this in mind as we consider what values are reflected in the model.

23

General Principles• The accountability system should be based on

multiple indicators to provide a more complete picture of schools• But should not overwhelm schools’ data capacity

• The measures should be combined using a compensatory framework to improve the reliability and validity of the decisions

• Every group should be required to meet an acceptable level of status or progress, but not both

24

Subgroups

• The same approach for holding subgroups accountable (i.e. the “super-subgroup”) in the 3-8 system will be used in the high school system as well

25

A Stratified Approach

• Long-term plan that focuses on valuing the “most important” things that high schools should be doing

• Short-term plan that addresses things that can be measured fairly across schools and not create a huge data burden

26

The Short-term reality• While the long-term system is

appealing, we are probably several years away from implementing such an approach

• We need to find a short-term solution that meets the requirements of the law, supports the long-term view, is fair, and can be implemented in 2007

27

Short-term proposal

• This approach focuses on a specific “checkpoint” to identify schools most in need of assistance to meet state standards

• Participation, growth, and status should be required components of the system

• Subgroups must be held to the same standards as the full school

28

Participation

• Will be calculated using:• UBSCT--10th graders attempting all 3

subtests/all 10th graders

• CRTs—all CRT test takers/all students enrolled in CRT courses

• DWA—all students tested/all 9th graders

Agreed!!!!

29

HS Status Components• 10th grade UBSCT scores in—reading, writing,

math• CRTs—counting all those scores earned by :

• ELA--9, 10, 11• Math CRTs—Algebra, Geometry• It was suggested including all math CRT scores

whenever they take it• Science—Earth Science, Biology, Chemistry, Physics

• All 9th grade Science CRT would count• All 10th grade Science CRT would count• Only the 11th grade CRT for students who did not take the

science CRT in both 9th and 10th. (For the kids who have not received the 2 credits by 11th grade, their 11th grade science CRT will count)

• Denominator= 11th graders and below with 2 CRT credits or fewer

30

HS Status Components• Math Courses—full year credit for 9th and 10th

grade students enrolled in a non CRT course• Students earning credit/total number of students

enrolled in non CRT classes in 9th and 10th grade• What is a non CRT course? (Concurrent enrollment,

Remedial math, CTE courses?)

• DWA—9th grade• UAA—in 9th and 10th grades• Graduation/completion rate• Attendance (18 or more absences/year)????

• The 3-8 system has attendance as 15 or more absences per year.

31

UBSCT Status

• The denominator will be all 10th graders in the school

• The numerator will be the count of students passing the first administration

32

Graduation rate• Same calculation method as AYP• Will move to a longitudinal system when the data system

allows for it

• What are the targets for graduation?• Same as AYP?• Will this simply be incorporated into the status calculation? If so,

what metric will we use?• For example, given the current status calculation plan a difference in

grad rates between 80 and 95% might NOT have a noticeable effect on overall status.

• Should graduation rate be a separate, conjunctive indicator?

Graduation rate was not discussed – we ran out of time.

33

Status Computation Questions

• How do we want to weight the various status components?• All content areas weighted the same?• Do math courses count the same as CRT

scores?• Graduation? Separate or incorporated?• UBSCT weight

This was not discussed – we ran out of time

34

Growth components• ELA--longitudinal growth from grades 8-11• For Science and Math:

• Use 8th grade scores as a pretest for the 10th grade CRT

• For UBSCT:• Use the same subject CRT score in 8th grade as a pretest

score for the UBSCT subject area test and measure student longitudinal growth

• These would all be measured using a value table approach similar to the 3-8 assessments

• We could build a UBSCT improvement statistic—evaluate the difference between initial pass rate and ultimate pass rate

• This was not discussed – we ran out of time

35

U-PASS High School Next Steps

Committee Meetings will continue

36

AYP

• New Amendment• An LEA is identified for improvement only

when it misses AYP in the same subject and in all grade spans for two consecutive years, or the other academic indicator in all grade spans for two consecutive years.

• New Report

37

Standard Testing Administration

Testing Ethics

• What are the concerns?• Lack of Training?• Inappropriate behavior?

• What are the solutions?

38

District

Sharing

39

New

Accommodations Policy

40

Utah Academic Language Proficiency

Assessment

UALPA

41

Proficiency Levels

A P Pre-Emergent

B E Emergent

C I Intermediate

D A Advanced

E F Fluent

42

Students to be assessed

• All P (pre-emergent) students (A)

• All E (emergent) students (B)

• All I (intermediate) students (C)

• No A (advanced) students (D)

• No F (fluent) students (E)

• No Native Speakers tested

43

Students to be assessed

• Students double tested to bridge scores from IPT to UALPA

• The same student must take the IPT and the UALPA in the same time frame

Minimal Requirement

• Double test 8% of P (A) students

• Double test 3% of E (B) students

• Double test 5% of I (C) students

44

Placement Test

• IPT used for Placement 06-07 year

• UALPA used for Placement in 07-08 year?

45

Test MaterialsStudent Booklet

• 1 booklet for each grade span• K - flip chart

• Speaking – flip chart • One flip chart for 1 – 6 • One flip chart for 7 – 12

• 1-2 consumable booklet

• 3-6, 7-8, 9-12 student booklet• Booklet includes listening, reading, writing

46

Test Materials

Answer document for each student

• K • Teacher completed

• 1-2 consumable booklet

• 3-6, 7-8, 9-12 • Speaking – teacher completed• Listening, reading, writing - Student completed

47

Student BookletGrade Span Speaking Listening Reading Writing

K Flip chart

1-2

Flip chart

Consumable booklet

3-6 Student booklet

7-8

Flip chart

Student booklet

9-12 Student booklet

48

Answer DocumentGrade Span Speaking Listening Reading Writing

K Teacher completed

1-2Teacher completed

(as part of cons. Booklet)

Consumable booklet

3-6Teacher completed(as part of student

document)Student Document

7-8Teacher completed(as part of student

document)Student Document

9-12Teacher completed(as part of student

document)Student Document

49

Test Materials

• Test Administration Manual (TAM)• Includes basic training information• Includes all administration information

• Examiner script

• Includes scoring guides for • Speaking – teacher completed• K – teacher completed

• Coordinators Manual • Test distribution information• Test return information• Scoring information

50

TEST DESIGN

51

Test Design Proposal2007-08

Test Design

Pre-Emergent Emergent Intermediate Advanced

Form A-B 

 Form B-C

52

Student Report Card

53

54

Scoring for Student Written Responses

• 2006-07• All student written responses for reading

and writing will be scored by contractor

• 2007-08• All student written responses for reading

and writing will be scored at the local level

55

• Yet to be finalized• Writing scoring guide

• By December 2006

• Process & Training for Districts to score writing in future years

• By May 2007

56

Training• Administration Manual

• Provides all training materials (2 pages)

• State Training Meeting• 2 Meetings - 2 choices

• October 5 (Utah Valley?)• October 17 (Salt Lake Valley?)

• Train the trainer model• Training Power-point provided• Training for scoring speaking

57

Timeline• July

• Item Selection & Edits• Test Direction Edits• Form Construction begins

• August & September• Form construction completed• All Test Materials completed• Test printing completed• New RFP

58

Timeline

• October• Train the Trainer Workshops

• Testing materials distributed to districts as printing completed.

• Pre-print file to Sharon by October 15• All testing materials distributed by

October 30

59

Timeline Options

• March• Score all tests administered by end of Feb.

• Minimum of 50% students assessed• All double testing completed• Measured Progress

• April• Standard Setting & Scoring• Test Results

• May• Score remaining tests• Test Results

60

AMAO Accountability

• New Report

• All data taken directly from USOE• Contractor and Computer Services do all

scoring

61