18
Maximizing Evaluation Impact by Maximizing Methods: Social Network Analysis Combined with Traditional Methods for Measuring Collaboration Carl Hanssen, PhD & MaryAnn Durland, PhD American Evaluation Association Baltimore, MD November 7, 2007

Maximizing Evaluation Impact by Maximizing Methods:

  • Upload
    damara

  • View
    65

  • Download
    0

Embed Size (px)

DESCRIPTION

Maximizing Evaluation Impact by Maximizing Methods:. Social Network Analysis Combined with Traditional Methods for Measuring Collaboration Carl Hanssen, PhD & MaryAnn Durland, PhD American Evaluation Association Baltimore, MD November 7, 2007. Agenda. Social Network Analysis: The Method - PowerPoint PPT Presentation

Citation preview

Page 1: Maximizing Evaluation Impact by Maximizing Methods:

Maximizing Evaluation Impact by Maximizing Methods:Social Network Analysis Combined with Traditional Methods for Measuring Collaboration

Carl Hanssen, PhD & MaryAnn Durland, PhDAmerican Evaluation AssociationBaltimore, MDNovember 7, 2007

Page 2: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 2

Agenda

Social Network Analysis: The Method

SNA Results and Interpretation

Next Steps

Page 3: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 3

SNA Methodology

Network Analysis is the study of the relationships formed by the interaction or links between components in a “set”.

MMP sets are schools The components are individuals:

Faculty, both math and non math MTL (School level Math Teacher Leaders) MTS (District level Math Teacher Specialists)

Relationship is communication about math

Page 4: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 4

Measures

Indegree – popularity Density – how “thick”, how much,

out of potential

Page 5: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 5

Role in Evaluation

How much does the communication structure actually fit the theory and the design of the project

Can the structure be correlated with other measures of implementation and impact? Activities Proximal measures

Page 6: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 6

MMP Evaluation Logic ModelStudent

Achievement

Teacher Content& Pedagogical

Knowledge

Math FacultyInvolvement

Learning TeamEffort

SchoolBuy-in

TeacherInvolvement

NewCourses

DistrictBuy-in

MPA Ownership

MATCBuy-In

UWMBuy-In

ClassroomPractice

MMPActivities

ProximalOutcomes

DistalOutcomes

Page 7: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 7

MMP Report Card Indicators

19 indicators in 7 domains derived from in-school data collection, online surveys, and MPS data

1. MTS Assessment2. Collaboration3. Learning Teams4. Classroom Practice5. Professional Development6. Teacher MKT7. Student Achievement

Page 8: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 8

SNA In Context: Evaluation Results

StudentAchievement

Teacher Content& Pedagogical

Knowledge

Learning TeamEffort

SchoolBuy-in

TeacherInvolvement

ClassroomPractice

WKCEMean % Proficient = 44%

Overall rating = 3.5Gap MTL v. other teacher = .2Teacher Engagement = 3.2

Overall IRT = -0.34Algebra IRT = -0.18

Team Functioning = 3.5MMP Principles = 3.6LT Quality = 3.1

PD Hrs. = 17.8Facilitation Hrs. = 1.0PD Quality = 3.1

Network density = 6.7% / School density = 17.6%MTL Role = 13.8 / MTS Role = 5.3

SR MTL Engagement = 4.4 / MTS Quality = 3.0

MTS Assessment = 38.3 of 55

Page 9: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 9

Data Collection

Math stakeholders in each school were asked to name individuals with whom the communicated about mathematics

Statistical analysis focused on1. Network and in-school density2. Importance of MTL and MTS

Page 10: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 10

MMP Impact Continuum

Low High

Loose WebMTL Not CentralFew Links to MTLMTS OutsideFew Links to MTS

Tight WebMTL Central

Many Links to MTLMTS Inside

Many Links to MTS

Page 11: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 11

LowSchool n Total Named

Network density

Density in school

MTL Role--In Degree

MTS Role--In Degree

G 17 40 6.2% 9.0% 17.31 3.85

Average 21.1 54.0 6.7% 17.6% 13.81 5.31 SD 6.8 17.6 2.6% 9.6% 7.2 4.9 Median 19 48 6.2% 15.4% 13.07 3.75

Page 12: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 12

MediumSchool n Total Named

Network density

Density in school

MTL Role--In Degree

MTS Role--In Degree

I 28.0 75.0 4.0% 12.2% 23.31 0.33

Average 21.9 57.1 6.3% 12.2% 18.84 2.69 SD 8.0 16.7 2.6% 5.0% 6.90 3.70 Median 22.0 51.0 5.7% 11.4% 17.56 0.92

Page 13: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 13

HighSchool n Total Named

Network density

Density in school

MTL Role--In Degree

MTS Role--In Degree

B 23 55 11.4% 31.1% 28.24 18.52

Average 21.1 54.0 6.7% 17.6% 13.81 5.31 SD 6.8 17.6 2.6% 9.6% 7.2 4.9 Median 19 48 6.2% 15.4% 13.07 3.75

Page 14: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 14

Student Achievement & In-School Network Density

Page 15: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 15

Student Achievement & MTL In Degree

Page 16: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 16

Conclusions

Distributed leadership—a key program goal is manifested by a tightly webbed network

School-level adoption of program principals is manifested by positioning of key individuals within the network

There may be a natural evolution of school networks that is indicative of program impact in that school

Page 17: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 17

Next Steps Continue school-level analysis to

strengthen our hypothesis about the relationship between social networks and other proximal and distal outcomes

Develop cross-school (or district-wide) networks

Page 18: Maximizing Evaluation Impact by Maximizing Methods:

11/07/2007 18

Contact Information

Carl Hanssen, PhDHanssen Consulting, [email protected]

MaryAnn Durland, PhDDurland [email protected]