Informing Reform: Changing The Relationships Between Research,

Policy and Practice in Context

Some Observations from the Work of the Consortium on Chicago School Research

Anthony S. BrykFounding Director

Reframing the relationship between . . .

Research

Practice Policy

A distinct mission —informing reform—anchored in a set of core values about the role of applied inquiryin social problem solving

Value Perspectives

• The relationship of social science to social problem solving in a democratic society is best viewed as an educational one. . . .In the broadest sense, we aim to educate the public about the improvement of public education.

Value Perspectives

• We eschew the role of “technical advisor to the prince.” We embrace a more democratic vision of policy making as a competition of ideas, where our role is to inform the competition with the best possible evidence and ways of thinking about that evidence.

Value Perspectives

• We embrace an ethical responsibility to seek out actively the diverse interests operative around school reform, and assure that a wide range of questions are continually addressed. The research agenda should not be beholden to any one group.

Value Perspectives

• A steadfast commitment to technical quality undergirds the conceptualization, design, and actual conduct of the research. While ideas from various stakeholders infuse the process; in the end we engage in disciplined inquiry.

Value Perspectives

• We are mindful that instrumental use of research findings is at best occasional. Distinctive contributions can also be made by reshaping the larger conceptualizations that focus subsequent policy debates—the importance of an empirically grounded, coherent theory of action.

Five Key Organizational Features

1. Extensive stakeholder consultation on research agenda, study design and interpretation of results

2. Distinctive model of funding and governance

3. Building analytic capacity for the long term

4. A commitment to intensive public informing

5. A distinctive researcher role within a network organizational structure

1. A Leadership Structure Designed For Consultation

Directors from multiple institutions• Responsible for day-to-day research

Steering Committee brings diverse perspectives• Establishes agenda of supported research

• Provides intellectual guidance

• Insures multiple perspectives

Constituent Advisory Board • Build a civic-elite constituency for the work

• Advance use of information

Ad hoc committees for particular studies

• Formally, Directors have extensive control over the work, but…

• Operate in an extraordinarily public and deliberately pluralistic environment

– role of Steering Committee in strategic resource allocation

– role of both Steering Committee and Constituent Advisory Board in shaping and reviewing work in progress

– all data made public

• General support from three major foundations (plus project specific support)

• Accountability

– on regular basis to Steering Committee and Constituent Advisory Board

– when requesting grant renewal

2. A Distinctive Model of Funding and Governance

3. Building Analytic Capacity for the Long Term

• Developing a base of researchers focused on reform in a place

• Enhancing the capacity of the system to understand its own operations

• Expanding the analytic resources for this work—a comprehensive longitudinal archive on students, schools and communities.

Multiple Methods and Measures

• Surveys

• Longitudinal case studies of schools

• Short-term case studies

• Analysis of administrative records

• Classroom observations

• Collection of student work

• Network analysis

• Longitudinal analysis of test scores

Data Source Level(s) Content Illinois State Report Card State, District, School Demographic characteristics,

achievement, truance, etc. Principal Survey (1992) School School reform, governance, program

participation, satisfaction Teacher Survey (1991) Teacher, School School reform, governance,

instructional practices, satisfaction Teacher Personnel Data (1993, 1997, 1998 forward)

Teacher Education, certification, tenure

Teacher Survey (1994) Teacher Instructional practices, teachers’ work lives

Student Survey (1994) Student Engagement, extracurricular activities, attendance, classroom experience

Principal, Teacher, and Student Surveys (1997, 1999, 2001.2003,2005)

Principals, Teachers, and Students

Five Essential Supports indicators and special reform topics.

Citywide Tests of Basic Skills (1987-2005)

Student Item responses and Rasch measures for reading and math

Administrative History Records Student Student mobility, high school course taking and failures, background characteristics

U.S. Census Data (1990 and 2000) Census Tract, Census Block Group

Mobility, language use, education levels, housing, etc.

Project on Human Development (1996, 2000 )

Community Institutional, social, and human resources.

Miscellaneous Other Community Data

Community (e.g. crime statistics, birth records, EPA sites, etc.)

A Comprehensive Data Archive

A Distinctive Research Process

• An interplay of theory, past research, and stakeholder concerns guide the questions we ask.

• The crafting of a research report typically begins as descriptive. As analyses proceed, a small set of focal ideas emerge.

• In directing attention to these ideas, we help to deepen public understanding, focus debate about policy concerns, and create opportunities to build common ground in a pluralist environment.

Media Output Audiences Stakeholder Input

Researchers, other urban school systems

Chicago public-at-large

Local school communities

Business leaders, Advocacy & Community

groups

Key Decision-makers

ConferencesReportsWeb site

Media

School reportsCable TV

Briefings Reports and summaries

Briefings Report summaries

Research Panels

Intermediary Groups (CALSC, Principals’ Assn..)

Constituent Advisory Board

Steering Committee

4. Intensive Public Informing

5. A Distinctive Researcher Role within a Network Organizational Structure

Sorting Through the Murky Terrain of Multiple Realities

• Individuals view schools through particular lens, according to interests, position and background

• Each brings insights that suggest questions and evidence to be gathered

• Extraordinary diversity of questions are possible

Researchers’ Distinctive Role in Pluralistic Environment

• Bring coherence and scientific discipline

• Keep research questions centered on fundamental issues that materially affect learning

• Reframes social and organizational working relationships

– Researchers have horizontal rather than vertical orientation

– Involvement of local researchers expands analytic capacity focused on schools

– Emphasis on mutual learning among researchers, policy makers and practitioners

• Stimulates broader processes of community education

• Advances conversation about school problems and how to proceed

Transforming the Researcher’s Role to Work within a Local Context

Challenges For Researchers

• Involvement of stakeholders complicates and lengthens the research process

• Exacerbates tension between timeliness and research quality

• Demands broader repertoire of skills among researchers

• Pressure on senior researchers to communicate frequently with multiple audiences

• Less flexibility for researchers in pursuing “their own research program”

Challenges Working with a School System

• Mutual trust• Collaboration with the research and evaluation

unit in the school system• Flexibility in adjusting to changes in leadership• Keeping top management informed: no surprises• Bring to light issues that need attention, avoiding

ad hominem criticism

• An inherent tension in “criticizing” the system as we seek to increase public understanding of what is required to expand opportunities

Necessary conditions: