Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education

Inside the ‘Black Box’ Of School Improvement: Measuring Change

James P. Spillane

Northwestern University

Chicago

March 11, 2008

Institute of Education Sciences,Spencer Foundation, National Science Foundation, Carnegie Foundation

www.distributedleadership.org

Organizing for Teaching and Learning

• Formal Structure

• Formal positions• Organizational routines

• Relational Structure: A Social Network Approach

• Norms and Beliefs• Network Structure• Substance of Interactions

Managing and Leading Instruction

• Heroics of Leadership• Principal Plus• Styles• Behaviors

• The missing piece - the practice of leading & managing

• A Distributed Perspective

• Attention to both formal and informal leaders• Practice is about interactions, not just actions

• The Subject Matters

Research Instruments

• Logs• Experience Sampling Method (ESM) log• Leadership Daily Practice (LDP) Log• End of Day Log (EOD)

• Social Network Survey

• Mixed Method Studies

Today’s Talk

• Motivating the work: Why Look Inside Schools?

• Social Network Survey Design & Validation

• Organizational ‘measures’• Norms & beliefs• Structure• Substance

Motivating the Work

Why Look Inside Schools?

Motivating the Work

• ‘Mediator Variables’

• “factors that must be changed or modified first by the treatment before main outcomes can be affected” (Petrosino, 2000, p. 50).

• “generative mechanism through which the focal independent variable is able to influence the dependent variable of interest” (Baron & Kenny, 1986, p. 1173).

• Build and test program/intervention theory (Weiss,1997).

• Hypotheses generating/theory development

Anchoring the Work: What to Measure

• First Dimension: Normative Structure• Social Trust• Collective Responsibility• Innovation

• Second Dimension: Relational Structure• Tie strength• Tie span

• Third Dimension: Substance of Interactions

Social Network Instrument

Design and Validation Work

Social Network Survey

Screen shot from SSSNQ Version 1 - Math Advice Questions

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.


Screen Screen Shot from SSSNQ Version 2 – Math Advice Questions Page 1











School staff network survey

M/R R/MRandomization

math name generator

math name interpreter

RWLA name generator

RLWA name interpreter

RWLA name generator

RWLA name interpreter

math name generator

math name interpreter

Three ways in which network data is susceptible to bias

• How many alters are named– fatigue or satisficing

• Who is named– censoring or priming

• Why those names were chosen/alter attributes– question scope redefinition

Average number of alters listedBy subject area and treatment group

Subject areaR/M

(n = 126)M/R

(n = 138) Difference

RWLA 3.2 1.9 1.3*

Math 1.5 1.7 -0.2

Total 4.7 3.6 1.1*

Some Results

The Instrument in Use

The First Dimension: Norms and Beliefs

• Why should we care:– Trust moderates sense of vulnerability and uncertainty that

accompanies change.– Trust enables joint problem solving.– Trust among school staff is linked to gains in student

performance.

• The Evidence from the field– Teacher-teacher trust– Teacher-principal trust

F A D C E G J I B H Total CCSRSchools

CCSRTeachers

strongly agree

strongly disagree

agree

disagree

Trust among teachers

M O L D A R K N E S B F H I C J G P Q Total CCSRSchools

CCSRTeachers

Trust among teachers

strongly agree

strongly disagree

agree

disagree

The Second Dimension: Relational Structure

• Frequent and influential interactions enable transfer of tacit, complex knowledge and joint problem solving.

• Relationships that span the formal organization allow for new ideas and minimize group think.

• Relationships that encompass all school staff members affects the degree to which staff members are engaged in joint work, and the consistency of advice provided.

Measuring Relational Structure

• Tie Strength

– Average number of advisors– Average number of advisors with at least weekly contact – Average influence

• Tie Span

– Average number of advisors in other grade level teams/departments– Average number of advisors external to the school

• Network Concentration

– Percent of ties to formal leaders– Distribution of advisors by number of advisees

Color corresponds to staff role

Reading/Writing/Language Arts

Math

Science

Social Studies

Special Education

6th Grade

Special Subjects

Administration

A

BC

Relational Structure



Math

Science

Social Studies

Special Education

6th Grade

Special Subjects

Administration

Relational Structure

M2

MathCoordinator

M2

M2

Average in-degree: M2 associates vs. Non-M2 associates

8.4

2.1

7.0

0.3

5.3

0

1

2

3

4

5

6

7

8

9

6th Grade Math Other

M2

Non-M2

N=6 N=95 N=16 N=35 N=541

• In math network, 8 out of 10 schools have two 6th grade team members connected to math department

Spanning Sub-units



Math

Science

Social Studies

Special Education

6th Grade

Special Subjects

Administration

• In RWLA network, 7 out of 10 schools have zero or one 6th grade team member connected to RWLA department

Spanning Sub-units



Math

Science

Social Studies

Special Education

6th Grade

Special Subjects

Administration

• In about half of the schools, an M2 associate provides the bridge between the 6th grade team members and the math department

Spanning Sub-units



Math

Science

Social Studies

Special Education

6th Grade

Special Subjects

Administration

M2M2

M2

M2

coordinatorM2 Math

Average external out-degree: M2 associates vs. Non-M2 associates

2.10

0.510.69

0.08

1.00

0

0.5

1

1.5

2

2.5

6th Grade Math Other

M2

Non-M2

N=5 N=86 N=16 N=29 N=445

M2 participants seek advice from more sources outside of their schools compared to their colleagues, suggesting that the M2 participants may be key brokers between their schools and external sources.

How is the red group different from the blue group?

• In some schools, one or two key players provided practically all of the advice (a high network concentration).

• In other schools, many teachers were involved (a lower network concentration).

• Generally, math networks were more concentrated than RWLA networks.

Structure: The Distribution of Influence

0

2

4

6

8

10

12

14

0

2

4

6

8

10

12

14

more concentrated less concentrated

• In some schools, one or two key players provided practically all of the advice (a high network concentration).

• In other schools, many teachers were involved (a lower network concentration).

• Generally, math networks were more concentrated than RWLA networks.

Concentrated Influence?

0

2

4

6

8

10

12

14

0

2

4

6

8

10

12

14

more concentrated less concentrated

M2 Math teachersM2 Math teachers

Self-contained teacherMath teacher

M2 Math teacher/coordinator

The Third Dimension

– The Substance of interactions among staff -- what teachers talk about!

– Instruction is a multi-faceted practice including • Deepening content knowledge • Course content • teaching strategies, • Materials• Strategies for low performing students• Assessing students understanding

0

1

2

3

4

5

6

O B H I M L C G F S D R P N A Q K J E Total

Less than Weekly Contact

Daily or Weekly Contact

RWLA NetworkAverage out-degree by school

Math NetworkAverage out-degree by school

0

1

2

3

4

5

6

M O C L J D B R S G H I Q P N A K F E Total

Less than Weekly Contact

Daily or Weekly Contact

Math NetworkPercent of ties that deal with each content area

0%

10%

20%

30%

40%

50%

60%

70%

80%

Deepening yourcontent knowledge

Planning orselecting course

content andmaterials

Approaches forteaching content to

students

Strategiesspecifically to assist

low-performingstudents

Assessing students’understanding of the

subject

Other

District Average

Individual School

www.distributedleadership.org

The instructional practices and assessments discussed or shown in these presentations are not intended as an endorsement by the U.S. Department of Education.

Documents

Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education