C R E S S T / U C L A 1

UCLA Graduate School of Education & Information StudiesCenter for the Study of Evaluation (CSE)

National Center for Research on Evaluation, Standards, and Student Testing (CRESST)

John J. LeeSenior Researcher

Quality School PortfolioDecision Support System:Transforming Data into Information

C R E S S T / U C L A 2

The Quality School Portfolio (QSP)

Without tools to investigate and understand results, more testing will not impact educational practice

Software for use by educators to organize, collect, interpret, and respond to complex data

UCLA, OERI, Stuart, and Joyce Foundation support

C R E S S T / U C L A 3

QSP—Making Use of Testing and Other Information

Data ManagerResource Kit

C R E S S T / U C L A 4

QSP Goals

Transform test and other results into usable, actionable form

Create coherent information structure for all members of the educational community (policymakers, administrators, teachers, and parents)

Encourage using information as the basis for making decisions about educational options and services

Raise capacity and efficacy of educators

Communicate findings in intuitive displays

C R E S S T / U C L A 5

Learning to Use Results: QSP Features

Relies on external and internal sources of information

Easy access and display

Custom queries

Disaggregation

Report preparation

C R E S S T / U C L A 6

Applying QSP

Use for school monitoring and improvement

For distance learning assessment reporting

For distributed program evaluation (LA’s BEST; Artful Learning)

Disaggregation

Individual longitudinal records

C R E S S T / U C L A 7

How can the school focus on particular populations?

System Groups

Custom Groups

Combo Groups

Disaggregation

C R E S S T / U C L A 8

Conditional Disaggregation: Eight Filters

Hispanic +

Boys +

9th Grade +

Spanish L1 +

Redesignated English

Magnet schools +

GPA below 2.0 +

Less than 2 years in the school

C R E S S T / U C L A 9

Gauge Display

C R E S S T / U C L A 10

Scatterplot

C R E S S T / U C L A 11

Quick School Comparisons

C R E S S T / U C L A 12

School Performance Reports

C R E S S T / U C L A 13

Resource Kit MenuResource Kit Menu

C R E S S T / U C L A 14

Resource Kit Functions

Clustered topics of resources

Selection or modification of instruments in a cluster given a description

Supports Acrobat Reader and Rich Text Formats

Customized entry of standard surveys, etc., to permit local use of evaluation data and simultaneous export of information to centralized sources

C R E S S T / U C L A 15

QSP Resource Kit

Local data to answer local questions

Staff and student surveys, interview and observation protocols

Data entry mechanisms, e.g., student work—in process (digital portfolios)

Online assessments or surveys

Transfer into Data Manager

C R E S S T / U C L A 16

Resource Kit Summary PageResource Kit Summary Page

C R E S S T / U C L A 17

Extensions

QSP Web (currently in development)

Data analysis from descriptive to growth projection

Flexible report wizard

Improved Web training component

C R E S S T / U C L A 18

User Challenges

Learning to ask productive questions

Looking at measures over time

Level of technical sophistication, e.g., hierarchical models, trajectories

Reports appropriate to various audiences and stakeholders

C R E S S T / U C L A 19

Continuing Research: A Sample of Impact Questions

Who is helped? What kinds of schools, teachers, and students? How do experience, beliefs, knowledge condition impact?

Do administrative practices change at school, district and board levels?

What happens in classrooms? Is student growth affected? For how long? What kind of combination of information is needed (CRT/NRT, process)?