Commissioner’s Performance-Based Accountability Task Force: System, Indicators, and Inputs Deb Wiswell, Keith Burke, & Scott Marion December 18, 2009

Commissioner’s Performance-Based Accountability Task Force: System,

Indicators, and Inputs

Deb Wiswell, Keith Burke, & Scott MarionDecember 18, 2009

Overview of Meeting

Commissioner's Performance Based Accountability System Task Force. 11/30/09

2

Brief overview of entire system Presentation of the “input” system Indicators and metrics A general approach for the performance

system

Adequacy Accountability System


Two approaches for demonstrating that the school is providing the opportunity for an adequate education Input System—Based on a subset of the school

approval standards Performance System—what this committee is

required to design

A school has to demonstrate that it is providing an opportunity for an adequate education by meeting the requirements of at least one of the systems, but not both.

The Input System


I. A school may demonstrate, through the input-based school accountability system, that it provides the opportunity for an adequate education as set forth in RSA 193-E:2-a by establishing that it met the following school standards in effect as of the effective date of this section: (a) English/language arts and reading as set forth in Ed 306.37.(b) Mathematics as set forth in Ed 306.43.(c) Science as set forth in Ed 306.45.(d) Social studies as set forth in Ed 306.46.(e) Arts education as set forth in Ed 306.31.(f) World languages as set forth in Ed 306.48.(g) Health education as set forth in Ed 306.40.(h) Physical education as set forth in Ed 306.41.(i) Technology education, and information and communication

technologies as set forth in Ed 306.42 and Ed 306.47.(j) School year as set forth in Ed 306.18.(k) Minimum credits required for a high school diploma as set forth

in Ed 306.27(f) and (m).

III. In order to demonstrate that a school provides the opportunity for an adequate education through the input-based school accountability system under paragraph I:


(a) The commissioner shall require school officials to submit a narrative explanation detailing how the school has complied with each of the standards included in the opportunity for an adequate education contained in paragraph I. The school principal and school district superintendent shall certify in writing that the responses submitted are accurate. The commissioner shall develop a form which conforms to the provisions of this paragraph.

(b) The commissioner shall review the responses to each school’s self-assessment required under this section and shall verify that the responses comply with the standards included in the opportunity for an adequate education specified under paragraph I.

(c) Schools that successfully demonstrate that they provide the opportunity for an adequate education through the input-based school accountability system for any year beginning with the 2009-2010 school year shall be required by the commissioner to resubmit the narrative explanations at least once every 2 years.

(d) Schools that are unable to demonstrate that they provide the opportunity for an adequate education through the input-based school accountability system for the 2009-2010 school year, or for any year thereafter, shall be required by the commissioner to resubmit the narrative explanations annually until such demonstration has been made.

(e) The commissioner shall integrate, to the maximum extent practicable, the input-based school accountability system to demonstrate the opportunity for an adequate education with the school approval process pursuant to RSA 21-N:6, V.

(f) Beginning September 1, 2012, the department shall annually conduct site visits at 10 percent of schools statewide to assess the validity of the input-based school accountability system and to determine whether those schools demonstrate the opportunity for an adequate education by meeting the school standards identified in paragraph I.

Demonstration of Input Data Collection


Note the use of the word “narrative” The AYP task force was concerned that unless this

was well structured, it would be almost impossible to score reliably and it would take too time much for school administrators to do well

Therefore, Keith Burke in collaboration with Deb Wiswell, content specialists, and the AYP task force, produced the input data collection software that we will now demonstrate

The purpose of this demonstration is to simply provide you with an overview of the types of indicators/measures being collected

You will all have an opportunity on the specifics of this approach, but NOT TODAY!

Switch to Input System demo


Back to the “performance system”


The discussion of potential indicators a few weeks ago was very interesting and made us think hard about how to respond to the task force’s input

We will propose a system later that we think can meet both the law and the interests of the task force (at least we understand it)

First, a few reminders of what is legally required….

SB 180 Task Force must… (a) Define the performance-based accountability

system to be used by schools that will ensure that the opportunity for an adequate education is maintained.

(b) Identify performance criteria and measurements. (c) Establish performance goals and the relative

weights assigned to those goals. (d) Establish the basis, taking into account the totality

of the performance measurements, for determining whether the opportunity for an adequate education exists, which may include the assignment of a value for performance on each measurement.

(e) Ensure the integrity, accuracy, and validity of the performance methodology as a means of establishing that a school provided the opportunity for an adequate education as defined in RSA 193-E:2-a.

SB 180 Requirements The task force shall develop a

performance-based scoring system using only the best available data and indicators which are already provided to the department and/or performance measures that schools are already required to provide the department under other state or federal law.

See handout listing already collected indicators

A little more on potential indicators


There seemed to be considerable interest in either non-NECAP-based indicators and/or process indicators

A recent analysis of other states suggests that few, if any, states are using such data in their school accountability systems.

As you can see on the following slide, no states are venturing into using the types of process indicators we discussed in November.

From Perie, et al. (2007)


California’s Academic Performance Index (API) includes both criterion-referenced and norm-referenced assessment results in ELA, math, science, and history/social sciences—different indicators are used at different grade levels.

Kentucky uses results from their science and social studies tests in their state accountability system. Also include a writing portfolio at grades 4, 7, and 12, test arts & humanities at grades 5, 8, and 11, and practical living & vocational studies at grades 5, 8, and 10.

Oklahoma and Ohio use additional indicators in their high schools that measure the percentage of students receiving advanced placement (AP) credit and taking the ACT or SAT.

Missouri uses the following indicators in their state accountability system in addition to the traditional achievement and participation measures: ACT (percent of graduates scoring at or above the national mean) Advanced Courses (percent of credits earned) Career Education Courses (percent of credits earned) College Placement (percent of graduates entering college) Career Education Placement (percent of career education completers

placed in occupation)

Process Indicators


What process and/or non-cognitive indicators are recommended?

Andy Porter, after surveying the school effectiveness literature, recommended a set of school quality indicators as a starting point for having schools focus on high leverage processes and indicators

Porter, Andrew C. (1991). Creating a System of School Process Indicators. Educational Evaluation and Policy Analysis, 13, 13-29.


Instructional leadership Existence of a focused mission High expectations for all students Task-oriented climate System for monitoring outcomes within the school School and class size Grade level organization Teacher planning time Availability, quantity, and credentials of teacher

aides Number of teacher preparations required Shared decision making

Except…


NHDOE is not in a position to collect data on most of the indicators on the previous slide

Further, NH schools are not in a position to provide systemic, accurate, and reliable data to the STATE on most of Porter’s indicators

Are these things important? Of course, but does that mean the state needs to

collect them? We would argue that the answer is a resounding “no”!

But should schools and districts be collecting such data? In this case, we would argue that the answer is “yes” IF the

local board and stakeholders believe the indicators are important

Indicators and Metrics


Indicators are the data sources such as the NECAP math testing

Metrics are the way that we quantify such indicators Status = A point-in-time measurement, e.g., 72%

of the students are proficient in math Improvement = Is generally the change in status

measures when students are NOT matched, e.g., 5% more students are proficient this year compared to last year

Growth = Is based on determining changes over time based on following the same students, e.g., our 5th grade students grew at a rate of X compared to where they were in 4th grade.

Potential NECAP Metrics


Status Growth Gaps

Status—e.g., are there significant differences in index values for students receiving free and reduced price lunch compared with those who do not receive free and reduced price lunch?

Growth—e.g., are there significant differences in growth rates for students receiving free and reduced price lunch compared with those who do not receive free and reduced price lunch?

An example from Colorado: Indicators in each of four categories

18

A Multi-Level Performance Accountability System for NH


Level One A very limited set of common (across the state)

indicators and metrics Level Two

Locally determined goals, targets, and indicators

Level One: State System


A very limited set of common indicators and metrics

Applied consistently across all schools in the state

Focused on unarguable outcomes, e.g., NECAP, graduation rate, postsecondary assessments, attendance

Level One: State System Indicators


All schools Test participation NECAP index scores (status)—reading, writing,

math, science NECAP status gaps—reading, writing, math,

science Attendance/truancy

Additional indicators for K-8 Schools NECAP growth percentiles (growth)—reading, math NECAP growth gaps—reading, math

Additional indicators for High Schools Graduation rate Postsecondary assessments (AP, SAT, ACT)?

Level Two: Locally-determined system


A very limited set (e.g., 2-5) of district/school-determined goals, targets, and indicators For example, “increase the % of students achieving their

NWEA growth targets to 90% by 2015” Could (should?) use the Follow-the Child framework Local Board approval of goals, targets, and indicators Local district would have to publish the goals and

indicators it is using AND the results of how it is doing on achieving these goals

NH DOE would provide guidelines on appropriate process and outcome indicators

NH DOE would have to approve the goals and indicators process (but not necessarily the specific goals)

Considerations


Level One—can meet the requirements of SB 180 on its own

Level Two—should it be required of all districts/schools or left optional? Theory of action supports requirement “unfunded mandate” argues for optional If optional, it all rests on Level One

How should the evaluations from Level One and Two be combined? Relatively equal weight? Level two as extra credit or tie-breaker?

Other considerations, concerns, questions, etc?

Documents

Commissioner’s Performance-Based Accountability Task Force: System, Indicators, and Inputs Deb Wiswell, Keith Burke, & Scott Marion December 18, 2009