Robert Barnoski (Barney)Robert Barnoski (Barney)

Washington State Institute for Public PolicyPhone: (360) 586-2744

E-mail: barney@wsipp.wa.govInstitute Publications:

www.wsipp.wa.gov

Promoting Academic SuccessEvaluation Progress

Washington Education Research Association Washington Education Research Association December 7, 2006December 7, 2006

Page 2 of 20

Washington State Institute for Public PolicyWashington State Institute for Public Policy

Created in 1983 by the state Legislature

Mission: Carry out non-partisan research on projects assigned either by the legislature or the Institute’s Board of Directors

8 legislators 4 higher education provosts or presidents 4 state agency directors

Page 3 of 20

2006 Legislature directed Institute to Evaluate Promoting Academic Success

The evaluation should focus on determining:(a) overall effectiveness in helping students pass the WASL;(b) relative effectiveness of different remedial strategies

offered; and (c) relative effectiveness by student characteristics, including

economic status, limited English proficiency, and ethnicity.

The office of the superintendent of public instruction shall provide all data necessary to conduct such analyses, and shall help coordinate data collection directly from districts administering the remedial programs as necessary.

ESSB 6386, §607 (11), Chapter 372, Laws of 2006, supplemental operating budget.

Page 4 of 20

An interim report is due to the Legislature by December 15, 2006.

The final report is due December 15, 2007.

Page 5 of 20

Analyses of 10th Grade Spring 2006 WASL Data

Reports on Website

www.wsipp.wa.gov

Page 6 of 20

(a) Evaluating Overall Effectiveness of Summer 2006 PAS

Compare WASL Retake Results for:• Students who participated in Summer 2006 PAS and retook

WASL in August, • With similar group of students not in PAS but who retook

WASL in August.

• Key is finding a similar comparison group.• Statistical control variables (covariates) are still used in

multivariate analysis to estimate impact of PAS participation on WASL retake.

• Two dependent variables: – Met standard and – Gain on WASL scale scores.

Page 7 of 20

(a) Evaluating Overall Effectiveness of Summer 2006 PAS

PAS Student Data Collection:

• OSPI built an application for recording students participating in PAS.

• Concerns about privacy kept OSPI from requesting the names of the teachers who provided the PAS instruction to the PAS students.

• As a result, Institute must rely upon the using district, school, content area, and PAS program name to link teacher surveys to student WASL results.

• Institute must reply upon school districts recording which students took summer PAS.

• OSPI called districts to encourage them to enter Summer 2006 PAS student data.

• School districts have until June 2007 to enter PAS students.

Page 8 of 20

(b) Evaluating Relative Effectiveness of Different Remedial Strategies for Summer 2006 PAS

Measuring Different Remedial Strategies:• Institute developed three surveys for PAS teachers after

discussions with OSPI staff (Reading, Writing and Math).• Two versions were created: paper and pencil, and online.• The teachers were to be surveyed at the end of the

summer 2006 effort.• We will analyze the associations between different

strategies and gains on WASL retakes for students who experienced the strategies.

• Statistical control variables (covariates) are still used in multivariate analysis when estimating impact of PAS strategies on the WASL retake.

Page 9 of 20

(b) Evaluating Relative Effectiveness of Different Remedial Strategies for Summer 2006 PAS

Remedial Strategy Data Collection:• OSPI built an application to record PAS program and

teacher contact information. • Institute must reply upon district PAS coordinator

recording the summer 2006 PAS program/teacher contact information.

• Not all districts entered these data before the end of summer school.

• School districts continued to enter Summer PAS data in August and September.

• Institute sent out teacher surveys as soon as we received the PAS teacher’s contact information.

• Institute hired WSU Social & Economic Sciences Research Center to do follow-up phone calling.

Page 10 of 20

Examples of Strategies on Math Survey for Summer 2006 PAS

What assessments were used to determine student math needs at the beginning of this instruction? (Check all that apply)

Diagnostic test(s) Classroom assignments WASL math strand results Quizzes/tests Homework assignments Teacher observation Standardized achievement tests WASL math scores None Other (specify):

Page 11 of 20

• PAS teacher survey response rate is approximately 65 percent.

Page 12 of 20

In your section(s), approximately what percentage of time did you spend in each of the following areas? (Must add to 100%)

__ Academic/vocational/career counseling__ General instruction for math__ Motivation__ Specific instruction on the EALRs and GLEs__ Study habits__ WASL test taking skills: guessing and scoring rules__ Other (specify):

Examples of Strategies on Math Survey for Summer 2006 PAS

Page 13 of 20

Examples of Strategies on Math Survey for Summer 2006 PAS

What instructional materials did you use in your section(s)? (Check all that apply)

OSPI Instructional Support Modules WASL practice tests/released items other than from OSPI Instructional

Support Modules WASL teaching materials other than OSPI Instructional Support Modules Materials aligned with EALRs/GLEs other than OSPI Instructional Support

Modules Materials specifically obtained for this remedial instruction other than OSPI

Instructional Support Modules District-adopted instructional materials Technology-based materials. Name(s): Other materials (describe):

Page 14 of 20

Examples of Strategies on Math Survey for Summer 2006 PAS

What special training have you received to teach students who have not met the 10th-grade math standards? (Check all that apply) OSPI Instructional Support Module training from OSPI staff OSPI Instructional Support Module training from district or ESD staff None Other training (describe):

In which of the following WASL math activities have you participated? (Check all that apply) Item writing Scoring Range finding Data review OSPI/ESD deep alignment Curriculum

alignment None Other (specify):

Page 15 of 20

(c) Estimating the Influence of Student Characteristics for Summer 2006 PAS

• Analyze association between different strategies and gains on WASL retakes by student characteristics.

Page 16 of 20

Number of PAS Teacher Surveys Summer 2006

Subject AreaSurveys Sent

Surveys Received

Percent Received

Math 346 223 64%

Reading 171 118 69%

Writing 181 116 64%

Page 17 of 20

Math Teacher Survey

Academic/vocational/career counseling1.9%

General instruction for math 46.7%

Motivation 9.2%

Specific instruction on the EALRs and GLEs 17.9%

Study habits 4.7%

WASL test taking skills: test taking, guessing, and scoring rules

17.3%

Other (specify): 2.3%

Total 100.0%

In your section(s), approximately what percentage of time did you spend in each of the following areas?

Page 18 of 20

Number of PAS Student RecordsSummer 2006

Subject Area

Records Received

Math 4,530

Reading 1,012

Writing 1,142

Total 6,684

Page 19 of 20

Number of WASL RetakesSummer 2006

PAS Reading

Spring 2006 Reading Level

Total Below Std Retake

% Retake

No L1 2,538 175 7%

No L2 6,716 1132 17%

No Total 9,254 1307 14%

Yes L1 149 71 48%

Yes L2 729 445 61%

Yes All 878 516 59%

Page 20 of 20

Number of WASL RetakesSummer 2006

PAS Writing

Spring 2006 Writing Level

Total Below Std

Retake Total

% Retake

No L1 2356 151 6%

No L2 8066 1464 18%

No Total 10422 1615 15%

Yes L1 168 78 46%

Yes L2 844 512 61%

Yes Total 1012 590 58%

Page 21 of 20

Number of WASL RetakesSummer 2006

PAS MathSpring 2006 Math

Level All Retake Total

% Retake

No L1 12828 1519 12%

No L2 15009 4208 28%

No Total 27837 5727 21%

Yes L1 1491 908 61%

Yes L2 2688 2036 76%

Yes Total 4179 2944 70%

Page 22 of 20

Thank You!Thank You!