Second Annual Race to the Top Professional Development Evaluation Report · 2015-12-03 · RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline March 2013 Consortium for

Second Annual Race to the Top

Professional Development Evaluation

Report Part II—Local Outcomes Baseline Study

Authors:

Jenifer Corn, Avril Smart, Lauren Bryant, Jennifer

Maxfield, Meredith Walton, Ruchi Patel, Sara Pilzer Weiss,

Brandy Parker, Shaun Kellogg Friday Institute for

Educational Innovation

Contributors:

Nate Barrett, Julie Marks, Tina Patterson, Carolina Institute

for Public Policy

March 2013

Consortium for Educational Research and Evaluation–North Carolina

Acknowledgements

We wish to thank Lynne Johnson, Maria Petrie-Martin, Neill Kimrey, Yvette Stewart, Eliz

Colbert, Robert Sox, and Stephen Jackson of the North Carolina Department of Public

Instruction for their time and assistance with data collection. We would also like to thank NCDPI

Division staff who led the professional development sessions we observed, as well as the

teachers and staff who participated in our site visits and focus group. We are grateful to Trip

Stallings and Glenn Kleiman from North Carolina State University’s Friday Institute for

Educational Innovation for valuable feedback on drafts of this report, and to Larissa Leroux for

editorial assistance.

RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline

March 2013

Consortium for Educational Research and Evaluation–North Carolina

Table of Contents

Executive Summary ........................................................................................................................ 3

Introduction ..................................................................................................................................... 7

Purpose of the RttT Evaluation and of this Report ...................................................................... 8

Overview of RttT Professional Development Activities and Supports Addressed in this Report .... 9

2011-12 NCDPI-RESA Sessions ........................................................................................... 10

2012 Summer Institutes .......................................................................................................... 10

Method .......................................................................................................................................... 12

Statewide Data Sources ............................................................................................................. 13

LEA Professional Development Coordinator Survey ............................................................ 13

RttT Omnibus Survey ................................................................................................................ 13

Longitudinal Sample.................................................................................................................. 14

School Sample Selection Process ........................................................................................... 14

Description of the Sample ...................................................................................................... 15

Longitudinal Data Sources and Procedure ................................................................................ 15

CLASS Observation Protocol ................................................................................................ 16

Teacher Professional Development Survey ........................................................................... 17

LEA Professional Development Coordinator Interviews....................................................... 19

Principal Interviews................................................................................................................ 20

Teacher Focus Groups ............................................................................................................ 20

Limitations ................................................................................................................................. 20

Results ........................................................................................................................................... 22

Evaluation Question 3.A.: To what extent did leader participants improve practices for support

of organizational change, including capacity to provide high-quality local professional

development aligned to RttT priorities? .................................................................................... 22

Planning, Accessing, and Implementing Local Professional Development........................... 23

Alignment with RttT Priorities ............................................................................................... 27

Quality of Local Professional Development .......................................................................... 29

Summary of Findings for Evaluation Question 3A ................................................................ 31

Evaluation Question 3.B.: How did school culture/organizational conditions change to support

RttT priorities? ........................................................................................................................... 32

Changes in Conditions related to Transition to the New State Standards .............................. 32

Changes in Conditions related to Data-based Decision Making and Revised Teacher

Evaluation Process ................................................................................................................. 34


March 2013

Consortium for Educational Research and Evaluation–North Carolina 2

Evaluation Question 3.C.: To what extent did teachers improve classroom practice? ............. 36

Next Steps for the Professional Development Evaluation ............................................................ 42

Continued Evaluation of the RttT Annual Professional Development Cycle ........................... 42

Continued Evaluation of the Distinguished Leadership in Practice (DLP) Principal Institutes ...... 43

Continued Collaboration with NCDPI Staff .............................................................................. 43

References ..................................................................................................................................... 44

Appendix A. RttT Professional Development Teacher Survey Items .......................................... 45

Appendix B. LEA Professional Development Coordinator Interview Protocol ........................... 50

Appendix C: Principal Interviews Interview Protocol .................................................................. 52

Appendix D. Teacher Focus Group Protocol ................................................................................ 54


March 2013


SECOND ANNUAL RACE TO THE TOP PROFESSIONAL DEVELOPMENT

EVALUATION REPORT, PART II—LOCAL OUTCOMES BASELINE STUDY

Executive Summary

The North Carolina Race to the Top (RttT) professional development plan is an expansive and

multi-faceted effort to increase student achievement by updating the knowledge and skills of the

state’s entire K-12 public education workforce. This initiative is driven by a host of recent

changes, including: adoption of new Common Core State Standards and North Carolina Essential

Standards; increased use of data to inform classroom and school decisions; rapid changes in the

technologies and digital resources available for teaching and learning; new teacher and

administrator evaluation processes; and an increased emphasis on formative assessment to

inform instructional decisions.

The human resources challenge of the initiative—to provide the state’s 100,000 teachers and

2,400 principals with professional development that will enable them to extend their knowledge,

improve professional practices, and, ultimately, increase student achievement overall and close

achievement gaps among student groups—is formidable. The timeframe (the four-year period of

the grant), diversity of the State (from large metropolitan local education agencies [LEAs] to

small, rural, and resource-limited LEAs, many of which continue to struggle under the weight of

fiscal constraints), and expectations (to create a statewide professional development

infrastructure that can be sustained after RttT funding ends) only increase that challenge. The

RttT professional development evaluation is being conducted in full recognition of these

circumstances, as well as of the deep commitment of the members of the RttT Professional

Development Implementation Team. The intent of the evaluation is to provide data-driven

information that can support reflection about and improvement of this effort.

Four general questions guide the evaluation:

1. State Strategies: To what extent did the state implement and support proposed RttT

professional development efforts?

2. Short-Term Outcomes: What were direct outcomes of State-level RttT professional

development efforts?

3. Intermediate Outcomes: To what extent did RttT professional development efforts

successfully update the NC education workforce?

4. Impacts on Student Performance: To what extent are gains in student performance

outcomes associated with RttT professional development?

The Evaluation Team is providing this second annual assessment of progress in three separate

but related reports. This report—the Local Outcomes Baseline Study— provides a baseline (first-

year) assessment of the State’s progress toward updating the education workforce in North

Carolina (Evaluation Question 3). For this report, the Evaluation Team (a) collected and

analyzed relevant data from all 115 LEAs, and (b) identified a purposeful sample of 27 schools

to examine more deeply the extent to which LEA and school staff increased capacity to provide

high-quality professional development. The other two reports address (1) the state’s progress in


March 2013


delivering face-to-face professional development statewide, and (2) implementation and impact

of the State’s Online Professional Development.

Summary of Major Baseline Findings

To address Evaluation Question 3, the Team examined the extent to which participation in the

state’s professional development efforts impacted (a) local capacity to provide high-quality

professional development, (b) shifts in local organizational conditions to support RttT priorities,

and (c) changes to instructional practice. Results from these baseline analyses will be used in

future reports to gauge progress toward updating the education workforce statewide.


of organizational change, including capacity to provide high-quality local PD aligned to RttT

priorities?

Planning and Accessing Professional Development.

For most schools, focus group and interview data reveal that school-level professional

development was informed by what was being offered at the state level and typically was

designed around resources available at the LEA level. In line with the state-endorsed “train the

trainer” model, LEA-level Professional Development Coordinators indicated that they were

primarily responsible for identifying and disseminating the most relevant professional

development resources and making them available to principals and teachers in their LEAs. Most

LEA Coordinators mentioned using tools like wikis, website postings, and staff development

newsletters and calendars to condense professional development resources and make them more

easily accessible to teachers and principals. However, despite the multiple methods of

dissemination described by LEA leaders, data indicate that some principals and teachers still

remained uninformed about NCDPI-support professional development opportunities.

Implementing Professional Development.

Most Professional Development Coordinators agreed that their LEA leadership had the capacity

to implement and plan professional development, but fewer believed that their LEA leaders used

data to inform their professional development decisions. Some LEAs called on teams of school-

level leaders to develop implementation strategies. In most schools, professional development

training was implemented through Professional Learning Communities (PLCs), and in some

LEAs, training opportunities were extended beyond the school year.

Alignment with RttT Priorities.

LEA Professional Development Coordinators statewide and teachers in the representative school

sample held different perceptions about whether professional development helped teachers build

their knowledge and skills related to some RttT priorities, such as understanding revised state

standards (91% of Coordinators but only 64% of teachers agreed that it helped) and deepening

their content knowledge (81% and 58%, respectively). For other priority areas, the perceptions of

both groups were more closely aligned, but were typically low (e.g., only 51% and 52%,

respectively, agreed that teachers built knowledge and skills related to revised state assessments).


March 2013


In addition, while 92% of Coordinators reported that LEA-provided RttT professional

development was aligned with and built upon existing professional development initiatives, only

64% of the coordinators agreed that their LEAs utilized data on staff technology proficiency

when planning RttT professional development. Furthermore, only 60% agreed that their LEAs

have plans for how to use the NCDPI Professional Development Leads in each region for

support in their delivery of professional development. Overall, these differing perceptions

suggest a lack of consistency among LEA PD coordinators in their efforts to align professional

development with RttT priorities at the local level.

Quality of Local Professional Development

As they reflected on the academic year, about 70% of teachers surveyed agreed or strongly

agreed that their local professional development experiences were both sustained and coherent,

though fewer believed that they had time to reflect on and experiment with what they learned—a

sentiment that was echoed in focus group responses. Teachers generally gave lower ratings for

their experiences with LEA-provided professional development than did LEA Professional

Development Coordinators, but interview and focus group data highlighted principals’ and

teachers’ positive experiences with local coaches and specialists, as well as their positive

impressions of the value of PLCs.

Evaluation Question 3.B.: How did school culture/organizational conditions change to

support RttT priorities?

Changes in Conditions related to Transition to the New Standards

Most LEA Professional Development Coordinators agreed that their LEAs have strategies in

place for communicating about availability of, collaborating on, and integrating state resources

into professional development related to revised state standards, but preparation for integrating

other aspects of the curriculum with those standards varies. Results from the RttT Omnibus

Survey showed that nearly all teachers felt strongly about their own preparation for the transition

to Common Core but were less sure about whether school leaders or other teachers in their

school really understood how to implement the new standards effectively.

Changes in Conditions related to Data-based Decision Making and the Revised Teacher

Evaluation Process

Between 2010-11 and 2011-12, there were slight decreases in teacher perceptions of their schools’

use of data for decision-making, as well as in their perceptions of the new teacher evaluation

process, but both changes may be the result of a sizeable increase in response rates. The Evaluation

Team will monitor annual survey results to determine whether these patterns persist.

Evaluation Question 3.C.: To what extent did teachers improve classroom practice?

Data gathered for this section provide baseline information about current learning activities for

comparison to similar data in subsequent years of RttT. Overall, Math teachers most frequently

reported daily instructional time on instructional strategies related to new standards, followed by

English Language Arts (ELA), Science, and Social Studies teachers. Across each content area:


March 2013


ELA: The highest percentage of teachers estimated that students spent time almost daily on

listening skills (76%) and on general vocabulary (66%).

Math: The highest percentage of teachers reported that on almost a daily basis, students made

sense of problems (80%), persevered in solving problems (72%), and used appropriate tools

strategically (67%).

Science: Teachers reported that their students practiced participating in hands-on activities

(32%), completed activities with a real-world context (24%), and used tools to gather data

(19%).

Social Studies: Students spent time almost daily on recognizing and appreciating

contributions of diverse cultural groups (30%), demonstrating chronological thinking (29%),

and analyzing cause-and-effect relationships (28%).

On average, observed classrooms in the sample schools received midrange ratings across the five

dimensions of the Evaluation Team’s standard classroom observation protocol1 that are most

directly aligned with implementation of the new Common Core and Essential Standards (Regard

for Student Perspectives, Quality of Feedback, Instruction Dialogue, Analysis and Problem

Solving, and Content Understanding). Across content areas, teachers sometimes scaffolded

discussion in the classroom and made attempts to integrate student feedback. ELA classrooms

were scored slightly higher on average than were other core courses in the frequency and quality

of examples of three of the dimensions of effective teaching (Regard for Student Perspectives,

Quality of Feedback, and Instruction Dialogue). Social Studies classrooms received the lowest

average ratings for observed frequency and quality of the other two dimensions (Analysis and

Problem Solving and Content Understanding). Finally, teacher surveys indicated that while some

formative assessment practices were common across classrooms, others were employed

infrequently. These observation data mirror results from teachers’ self-report surveys about

frequency of effective, standards-based teaching practices.

Summary of Results

Overall, the baseline results presented in this study indicate that schools and LEAs have taken

some initial steps toward developing a process for professional development that supports RttT

priorities, including transition to the new state standards, implementation of formative and

summative assessments, use of data to support instruction, and effective utilization of the new

North Carolina Educator Evaluation System. These aggregated results provide a cursory baseline

assessment of Year 1 progress toward achieving and sustaining local capacity to provide high-

quality professional development, supporting shifts in local organizational conditions to support

RttT priorities, and encouraging changes in instructional practice.

1 Classroom Assessment Scoring System, or CLASS (http://www.teachstone.org/about-the-class/). In all, there are

12 CLASS domains.

http://www.teachstone.org/about-the-class/


March 2013


Introduction

Providing high-quality, accessible professional development to all teachers and principals is a

key component of the professional development plan funded by North Carolina’s federal Race to

the Top (RttT) grant. The United States Department of Education’s grant application process

recognized the important role that professional development must play in the successful

implementation of the RttT education reforms by requiring states to develop comprehensive

strategies for both the expansion of their professional development offerings and the evaluation

of the effectiveness of that professional development. North Carolina’s RttT-funded professional

development plans are ambitious, with a top-level goal of updating the entire education

workforce to ensure that each of the state’s 100,000 teachers and 2,400 principals has the

knowledge and skills necessary to foster student achievement.

The challenges addressed by the North Carolina Department of Public Instruction’s (NCDPI)

RttT Professional Development Implementation Team are formidable. The Implementation

Team must develop strategies that, in coordination with Local Education Agencies (LEAs),

Regional Education Service Agencies (RESAs), and other organizations, will engage all of the

state’s teachers and administrators in professional development that will extend their knowledge

and guide them in making significant changes in their professional practices in order to increase

overall student achievement and close achievement gaps among student groups. The plan

requires differentiation of professional development to meet the needs of educators with a wide

range of backgrounds, preparation, experience, and expertise, ranging from the over 10,000

experienced National Board Certified teachers in North Carolina to the over 20,000 teachers who

are in their first three years of teaching and who in many cases entered teaching through an

alternative licensure pathway. The plan requires reaching teachers from kindergarten through

high school, across all disciplinary areas. The plan also requires professional development for

principals, assistant principals, curriculum specialists, and all of the other administrators

involved in guiding and supporting teachers through transitions to new standards, student

assessments, data systems, technologies, teacher evaluations, and overall expectations for both

themselves and their students. All of this is to be accomplished within the four-year period of the

grant, across a large and diverse state with many small, rural, and resource-limited LEAs. In

addition, the plan is expected to result in a statewide professional development infrastructure that

can be sustained after RttT funding ends. While the grant provides significant funding, the

professional development initiative must proceed during a time of severe economic constraints

that limit additional state and local resources that can be brought to bear on the challenges; this

limitation is far greater than was anticipated when the RttT proposal was developed.

The RttT professional development evaluation is being conducted with full recognition of the

enormous challenges being addressed by the Professional Development Implementation Team

and the deep commitment of the members of the team to do so. The Evaluation Team further

recognizes that the RttT professional development plan is breaking new ground: There are no

established, proven models for creating a professional development system of this scale to

support so many educators through so many changes in so short a period of time. The Evaluation

Team’s intent for this report is to provide data-driven information that can support reflection

about and improvement of the RttT professional development effort.


March 2013


Purpose of the RttT Evaluation and of this Report

The roles of the RttT Evaluation Team are to (1) document the activities of the RttT initiatives;

(2) provide timely, formative data, analyses, and recommendations to help the initiative teams

improve their ongoing work; and (3) provide summative evaluation results in Year 4 of the

project to determine whether the RttT initiatives met their goals and to inform future policy and

program decisions to sustain, modify, or discontinue initiatives after the grant-funded period.

The RttT Evaluation Team is documenting the professional development initiative activities and

collecting data about participation in, satisfaction with, and the impact of locally- and state-

supported professional development activities. In addition, the team is collecting information

through LEA and school field visits and teacher and administrator surveys to inform summative

evaluation analyses that examine the extent to which participation in RttT professional

development activities has increased the capacity of the education workforce to deliver effective

instruction. Ultimately, the evaluation will use state assessment data to gauge changes in student

performance. The full plan is described in greater detail in Appendix A.

Four general questions guide the overall evaluation effort, with a number of more specific

questions embedded in each one. These overall questions are organized to reflect the general

sequence in which they can be addressed over the four years of RttT:

1. State Strategies: To what extent did the state implement and support proposed RttT

professional development efforts?

2. Short-Term Outcomes: What were direct outcomes of state-level RttT professional

development efforts?

3. Intermediate Outcomes: To what extent did RttT professional development efforts

successfully update the North Carolina education workforce?

4. Impacts on Student Performance: To what extent are gains in student performance

outcomes associated with RttT professional development?

The Year 2 professional development evaluation serves three primary purposes, each of which is

addressed in a separate report, as described below. Each of the three reports is organized around

specific evaluation questions; this document is Report II of the Year 2 RttT evaluation.

Report I: RttT Statewide Face-to-Face Professional Development Formative

Evaluation—Report I documents the current status and direct outcomes of the state’s RttT

face-to-face professional development efforts in the NCDPI Annual Professional

Development Cycle described below, addressing questions under Overall Evaluation

Question 1 (State Strategies) and questions under Overall Evaluation Question 2 (Short-Term

Outcomes). It is intended to contribute to the RttT Professional Development Implementation

Team’s ongoing efforts to provide the most supportive and useful professional development

possible for teachers and administrators.

Report II: RttT Professional Development Local Outcomes Baseline Study—Report II

(this report) provides detailed information concerning near- and longer-term impacts of


March 2013


statewide face-to-face RttT professional development efforts at the local level. The

Evaluation Team collected and analyzed relevant data from all 115 LEAs and identified a

purposeful sample of 27 schools to examine more deeply the extent to which LEA and

school staff increased capacity to provide high-quality professional development and focused

on shifts in local organizational conditions to support RttT priorities, and changes to

instructional practice. That is, this section provides baseline data for Evaluation Question 3

(Intermediate Outcomes) and related sub-questions:

A. To what extent did leader participants improve practices for support of organizational

change, including capacity to provide high-quality local PD aligned to RttT priorities?

B. How did school culture/organizational conditions change to support RttT priorities?

C. To what extent did teachers improve classroom practice?

Report III: RttT Online Professional Development Evaluation Report—Report III also

addresses Evaluation Questions 1, 2, and 3 (State Strategies, Short-Term Outcomes, and

Intermediate Outcomes), but with a focus on all RttT-supported online professional

development. It provides detailed information about the extent to which the online

professional development components of the RttT proposal have been carried out, and the

immediate impact of RttT online professional development efforts on educators.

The next major assessment, scheduled for delivery in Fall 2013, will consider short-term

outcomes in more detail and will begin to address intermediate outcomes, while reports in

following years (after the local professional development programs are in operation) will address

the impact on teachers and, to the extent possible, students. Since the schedule of the Evaluation

Team’s deliverables calls only for annual reports, the team will actively coordinate with NCDPI

to share preliminary survey data reports with appropriate staff, in order to provide timely

information as the Professional Development Implementation Team continues to work with

LEAs and charter schools and develops additional resources.

Overview of RttT Professional Development Activities and Supports Addressed in this Report

As shown in the diagram of the Annual Professional Development Cycle provided by NCDPI

(Figure 1, following page), the overall professional development plan is built around annual

cycles comprised of Summer Institutes, formative support for LEA and charter school

Professional Development Leadership Teams, and additional face-to-face support sessions

provided by the NCDPI RttT Professional Development Implementation Team in collaboration

with RESAs. Through implementation of this Annual Professional Development Cycle, NCDPI

aims to both guide and support the development of local professional development through a

high-quality, systemic, blended approach to effective professional development, defined as job-

embedded, research-driven, data-informed, professional, community-based, and aligned to the

RttT initiatives. A blended approach to professional development is designed to address the

needs of LEAs, charter schools, and individual educators through face‐to‐face sessions and

online resources, such as learning modules or webinars.


March 2013


Figure 1. NCDPI Annual Professional Development Cycle

2011-12 NCDPI-RESA Sessions

NCDPI offered over 100 face-to-face NCDPI-RESA partnership sessions across the eight North

Carolina State Board of Education regions from August 2011 to June 2012. Sessions lasted

anywhere from three hours to two days and focused on Content Support (Common Core and

Essential Standards), Fidelity Support, Teacher Effectiveness/New Accountability Model, North

Carolina Professional Training Standards for Principals and Assistant Principals, Detailed

Scope of Work/Technical Assistance, Principal Training for Common Core and Essential

Standards, and Principal and Assistant Principal Training (ITES Standards). These sessions were

developed and facilitated by the NCDPI RttT Professional Development Implementation Team in

partnership with RESA staff.2

2012 Summer Institutes

Six two-day Summer Institutes were held across the state to provide opportunities for local

professional development leaders to expand their learning about the Common Core State

Standards and the North Carolina Essential Standards. LEA and charter school teams, comprised

of up to 18 members, including key leaders from each LEA and charter school, attended these

summer trainings. These sessions were conducted in collaboration with LEAs and facilitated by

2 For descriptions of these sessions, please see Table 10 in Report I (Report I: RttT Statewide Face-to-Face

Professional Development Formative Evaluation).


March 2013


the NCDPI RttT Professional Development Implementation Team. This year’s Summer Institutes

focused on how to facilitate learning for all students and included sessions organized by topic:

Content Strands, Leaders with Leaders, Facilitative Team Time, and Assessment and

Accountability.


March 2013


Method

In order to examine more deeply the extent to which LEA and school staff increased their

capacity to provide high-quality professional development at the local level, the Evaluation

Team collected and analyzed relevant data from all 115 LEAs (state strategies), and identified a

purposeful sample of 27 schools (short-term outcomes). Eight data sources were used over the

course of the 2011-12 academic year and the summer of 2012 (Table 1). The Evaluation Team

used a mixed methods approach to address the broad evaluation questions discussed above.

Quantitative methods included surveys and structured observations. Qualitative approaches

included interviews and focus groups, as well as open-ended survey questions. This section

discusses each of the data sources used and the corresponding methodology. Percentages

presented in the following tables have been rounded to the nearest whole number; therefore, not

all percentages add up to exactly 100.

Table 1. RttT Evaluation Data Sources, 2011-12

*RttT priorities include 1) transition to new state standards, 2) implementation of new formative/summative

assessments, 3) use of data literacy for instruction, 4) use of technology, and 5) effective use of the revised North

Carolina Educator Evaluation System.

Data source Purpose

Statewide

RttT LEA Professional

Development Coordinator

Survey

Gather information about local capacity for providing high-

quality professional development

RttT Omnibus Teacher

Survey

Assess change across a wide range of constructs that may be

influenced by the collective set of NC RttT activities

Longitudinal

CLASS Observation Protocol Gather information about classroom practices

RttT Professional

Development Teacher

Survey

Gather information about teachers’ attitudes, experiences,

knowledge, etc., about professional development and RttT

priorities*

RttT Professional

Development LEA

Coordinator Interviews

Discuss local capacity for providing high-quality professional

development

RttT Professional

Development Principal

Interviews

Discuss principals’ experience and recommendations about

RttT professional development related to RttT priorities*

RttT Professional

Development Teacher Focus

Groups

Discuss teachers’ experience and recommendations about RttT

professional development related to RttT priorities*


March 2013


Statewide Data Sources

LEA Professional Development Coordinator Survey

The Evaluation Team created and distributed a survey to LEA Professional Development

Coordinators to gain information on how the LEAs were developing and implementing

professional development programs that addressed the RttT initiatives, as well as whether they

were accessing and using NCDPI’s RttT resources and events. The survey items asked the

participants to use a 7-point Likert scale to rate their LEA in terms of how well their local

professional development efforts helped their teachers move toward RttT initiative goals (e.g.,

“Professional development experiences in my district this year have helped teachers implement

the state Standard Course of Study”). Participants also were asked to indicate the progress of

their LEA in building the capacity to continue serving their educators after the completion of the

RttT grant (e.g., “In my district, leaders have the skills to plan and design high-quality

professional development”).

The survey was distributed via email in February 2012 to LEA leadership with a request that it

be forwarded to the Professional Development Coordinator. Due to this deployment method, the

data were cleaned to remove any non-Professional Development Coordinators who may have

taken the survey. The Team received a total of 274 responses to the survey. After restricting the

data to those who specifically identified themselves as serving in a professional development

coordinator capacity, the final sample size used in the analysis was 159. Survey responses were

received from all eight regions, from 92 of the 115 LEAs (Table 2).

Table 2. LEA Professional Development Coordinator Survey Response Rate by Region, 2012

Region

Number of LEAs that

Completed the Survey

Number of LEAs in

Region Response Rate

1 13 15 87%

2 13 14 93%

3 12 14 86%

4 10 12 83%

5 10 15 67%

6 7 9 78%

7 16 19 84%

8 11 17 65%

Total 92 115 80%

RttT Omnibus Survey

The overall research plan for the four years of the NC RttT evaluation includes annual

administration of an Omnibus Teacher and Principal Survey to a statewide sample in February of

each school year. The survey was designed to assess change across a wide range of constructs

that may be influenced by the collective set of NC RttT activities, with items in several

constructs corresponding to specific initiatives, including professional development.


March 2013


The Omnibus Survey is comprised of 170 items across 23 dimensions. Examples of key

dimensions include teacher-leadership respect, teacher-teacher trust, teacher knowledge sharing,

and teacher-student relationships. Each respondent received a random sub-sample of the

questions to decrease respondent burden. The survey was administered online via Qualtrics, and

the sample of questions for each participant was estimated to take between 15 and 30 minutes to

complete.

For this report, five dimensions on the survey were identified to help understand the evolution of

the professional development component of RttT. The tables in this report were generated using

data from individual teacher responses to items in the following dimensions:

1. Quality of Professional Development

2. Alignment of Professional Development

3. Data-Driven Instruction

4. Evaluation of Teachers

5. Formative Assessment

For the first four of these dimensions, items were measured on a 7-point Likert scale: (1)

Strongly disagree, (2) Disagree, (3) Slightly disagree, (4) Neither agree nor disagree, (5) Slightly

agree, (6) Agree, and (7) Strongly agree. Items in the fifth dimension, Formative Assessment,

asked how often different formative assessment strategies were used; frequency was measured

on a 5-point scale: (1) Never, (2) A few times a year, (3) Once or twice a month, (4) Once or

twice a week, and (5) Almost daily. Since the nature of the questions and scale of the data were

different, the results for the Formative Assessment dimension are reported separately, after the

results for the first four professional development dimensions.

Longitudinal Sample

School Sample Selection Process

In order to gain a more accurate perspective on the impacts of the RttT professional development

initiatives on educators across the state, a purposeful sample of schools was selected to

participate in the longitudinal descriptive study. School type, professional development ratings,

and student achievement were the three primary variables that influenced the selection of the

schools. The Evaluation Team categorized all schools in the state by type (elementary, middle, or

high school), tercile of recent professional development quality (high, midrange, or low) based

on the results from the 2010 TWC survey, and tercile of school performance composite scores

(high, midrange, or low) based on the North Carolina ABCs of Public Education school

accountability model. School type, urbanicity, achievement status, teacher characteristics,

student characteristics, and other variables also were considered in the selection of schools for

the sample. Finally, considerations were made to ensure that the schools were geographically

representative. This selection process yielded a stratified sample of 27 schools.


March 2013


Description of the Sample

This section describes the sample schools in general terms; specific information that could be

used to identify individual schools was removed to prevent violation of confidentiality.

Schools. The longitudinal sample included nine traditional elementary schools (grades K-5),

seven middle schools (grades 6-8), nine high schools (grades 9-12), and two schools that blend

either middle and elementary grades or middle and high school grades. Across all 27 schools in

our sample, 83% of their Adequate Yearly Progress goals in the 2010-11 academic year were

met, and collectively, 75% of their students were at or above grade level. Approximately 19% of

schools were located in cities, 11% were suburban, 26% were in a town, and 44% were rural.

Schools were distributed across all eight State Board of Education regions.

Figure 2. Geographic Distribution of Longitudinal Professional Development Study Schools

Teachers and students. Across the sample schools, about 26% of teachers had advanced degrees,

18% were National Board Certified, and 96% were fully licensed. Eighteen percent of teachers in

the sample had three or fewer years teaching experience.

The ethnicities of students in the sample were comparable to those across the state (North

Carolina Department of Public Instruction, 2012). On average, about 55% were White/

Caucasian, 26% were Black/African American, 14% identified as Hispanic/Latino, 4% were

multiracial, 1% were Asian, and less than 1% were American Indian/Native American or Pacific

Islander. Approximately 55% of the students in the sample were eligible to receive free or

reduced-price lunch.

Longitudinal Data Sources and Procedure

During the 2011-12 academic year, the Evaluation Team gathered quantitative and qualitative

data on professional development activities in the 27 schools. Quantitative data were gathered

through structured classroom observations and teacher surveys. Qualitative data included semi-

structured interviews and focus groups with LEA Professional Development Coordinators,

principals, and teachers in the sample schools. All interviews and focus groups were facilitated

by Evaluation Team members. The following section describes each data source and the

procedures used to analyze the data collected.


March 2013


CLASS Observation Protocol

The Classroom Assessment Scoring System (CLASS) was used to gather data regarding

teachers’ progress in the transition to new state content standards (Common Core and Essential

Standards), which was a primary goal of the RttT-funded professional development. Research

has shown CLASS to be both valid and reliable (Center for Advanced Study of Teaching and

Learning, 2011), and it can be used in a wide range of classroom situations. In addition, the

tool’s depth offers several advantages over similar tools. CLASS offers different versions for

multiple grade levels, and its 7-point rating scale offers more rating flexibility and refinement

than do scales found in other tools considered by the Evaluation Team. All RttT evaluators who

conducted the classroom observations completed a certification process that consisted of two

days of training and successful completion of a CLASS Reliability Test.

CLASS was developed to assess classroom quality based on observed instructional interactions

between teacher and students, as well as productivity evident in classroom settings (Teachstone,

2012). The system has a total of 12 dimensions, but for this evaluation, the Team chose to focus

on the 5 CLASS dimensions that most closely align with instructional strategies outlined in the

Common Core and Essential Standards (Regard for Student Perspectives, Content

Understanding, Analysis and Problem Solving, Quality Feedback, and Instructional Dialogue;

Table 3). Each dimension was scored using the 7-point scale, with 1 being the lowest.

During the 2011-12 academic year, the Evaluation Team observed over 120 hours of teaching in

core content classrooms (English-Language Arts [ELA], Math, Social Studies, and Science)

within our 27 sample schools. Observers spent between one and two hours in each classroom in

both the fall and spring semesters.

Table 3. CLASS Observation Tool Dimensions Chosen for Use in the RttT Evaluation

CLASS Dimension Description

Regard for

Student

Perspectives

(RSP)

This dimension focuses on the extent to which the teacher is able to meet and capitalize on

the social and developmental needs and goals of adolescents by providing opportunities for

student autonomy and leadership. Also considered are the extent to which student ideas and

opinions are valued and content is made useful and relevant to adolescents.

Content

Understanding

(CU)

Content understanding refers to both the depth of lesson content and the approaches used to

help students comprehend the framework, key ideas, and procedures in an academic

discipline. At a high level, this refers to interactions among the teacher and students that lead

to an integrated understanding of facts, skills, concepts, and principles.

Analysis and

Problem Solving

(APS)

This dimension assesses the degree to which the teacher facilitates students’ use of higher

level thinking skills, such as analysis, problem solving, reasoning, and creation through the

application of knowledge and skills. Opportunities for demonstrating metacognition (i.e.,

thinking about thinking) are also included.

Quality of

Feedback (QF)

Quality of feedback assesses the degree to which feedback expands and extends learning and

understanding and encourages student participation. In secondary classrooms, significant

feedback may also be provided by peers. Regardless of the source, the focus here should be

on the nature of the feedback provided and the extent to which it “pushes” learning.

Instructional

Dialogue (ID)

Instructional dialogue captures the purposeful use of dialogue—structured, cumulative

questioning and discussion that guide and prompt students—to facilitate students’

understanding of content. The extent to which these dialogues are distributed across all

students in the class and across the class period is important to this rating.


March 2013


CLASS data were compiled for all classroom observation sessions during the 2011-12 academic

year, for a total of 208 observed classrooms (grades 1 through 12) and 702 observations in all.

The data were grouped by school performance level and by professional development rating (as

described above). Summary statistics for the various groupings were then calculated for the

classroom characteristics and the CLASS variables of interest listed above.

On average, classrooms contained one adult and 20 students. ELA represented over 30% of the

content observed in classrooms, followed by Math and Science, each of which represented 25%

of observed content. Twenty-one percent of classrooms taught Social Studies, and the remaining

focused on non-core content. Generally, classrooms observed for this sample used “whole class”

instructional formats for some or all of the instruction observed (69% of classrooms), followed

by “small group” and “individual instruction” formats (approximately 40% each). It is important

to note that because the study included elementary schools, Evaluation Team members had the

option to indicate observation of multiple content areas for each classroom characteristic per

observation. For example, in an elementary classroom, an observer may see both ELA and Math

taught, with the teacher using multiple instructional formats for each, all within the same

observation period.

Teacher Professional Development Survey

The Teacher Professional Development Survey (Appendix A) was designed to gauge teachers’

perceptions of the quality and utility of state- and locally-provided professional development

training. More specifically, items on the survey asked teachers to rate LEA leadership capacity

for providing professional development, the potential impacts of professional development on

teachers’ skills and knowledge in the classroom, and the integration of Common Core and

Essential Standards into classroom instructional practices. The items about instructional practices

aligned to the new standards were developed in close collaboration with NCDPI leaders in the

Curriculum and Instruction division for the core content areas.

Principals in each sample school disseminated the survey to teachers in their schools between

October 2011 and March 2012. Survey items included both Likert agreement scales and

frequency scales. Of the 1,211 participants who started the survey, 22 participants indicated that

they either did not teach a class (e.g., some participants indicated that they were counselors or

principals), were not certified to teach (e.g., teacher aides), or left the “certified to teach”

question blank; these participants were removed from the initial sample, reducing the sample to

1,189.

In general, teachers who completed the Teacher Professional Development Survey were

representative of teachers across the state. Similar to state-reported ratios in the Public Schools

of North Carolina statistical profile for the 2011-12 school year3 teachers in this sample were

75% female (n = 891), and over 80% were Caucasian/White (n = 890; Table 4, following page).

3 North Carolina Public Schools Statistical Profile, 2011–12

http://apps.schools.nc.gov/pls/apex/f?p=1:21:0::NO:::


March 2013


Table 4. Participant Representation by Ethnicity, Teacher Professional Development Survey

Ethnicity

Percentage of

Respondents

American Indian/Alaska Native 1%

Asian* <1%

Black/African American 13%

Native Hawaiian/Other Pacific Islander 0%

White/Caucasian 82%

Hispanic/Latino 2%

Multiracial 2%

*Representation for Asian is 0.2%

Table 5 shows the distribution of teachers in the sample across content areas. Overall, English

Language Arts (17%) and Math (16%) teachers were most represented among all the content

areas. As expected, 14% of elementary school teachers reported teaching all content areas.

Table 5. Content Areas Taught by Participants, Teacher Professional Development Survey

Content Area

Percentage of

Respondents

English/Language Arts 17%

Math 16%

Science 10%

Social Studies 10%

All core subjects (ELA, Math, Science, Social Studies)* 14%

Foreign Language 3%

Elective 14%

Other (e.g., JROTC, Business, Health) 26%

Note: Totals include teachers who did not indicate their school and therefore are not reflected in the professional

development categories.

*This category includes elementary school teachers who teach all four core subject areas; those teachers were

excluded from the counts of the individual core content areas.

Sixteen percent of teachers in this sample indicated that they held National Board Certification

for at least one content area in which they teach (n = 1,048). The majority of teachers reported

having a bachelor’s degree (34%), with 23% having a bachelor’s plus credits, and 25%

indicating they hold a master’s degree (n = 1,049 ; Table 6, following page).


March 2013


Table 6. Participants’ Education Experience, Teacher Professional Development Survey

Education Experience

Percentage of

Respondents

Bachelor’s degree 34%

Bachelor’s degree plus credits 23%

Certificate of Advanced Study 1%

Master’s degree 25%

Master’s degree plus credits 12%

Doctorate* 0%

I am a lateral entry teacher 4%

*Representation for those holding a doctorate is 0.5%

Response rate. Based on self-reported data from principals in our sample regarding the total

number of teachers in their schools, the overall survey response rate on the Teacher Professional

Development Survey was 77% (Table 7).

Table 7. Survey Response Rates, Teacher Professional Development Survey

School Level

Mean Response

Rate

Elementary School 83%

Middle School 56%

High School 87%

Overall 77%

Response rates for each question on the survey varied for a few reasons. First, participants were

able to skip questions. Second, many items offered the response option of “Not Applicable”; this

response option was treated as missing when calculating proportions of responses. Finally, the

response options for the Quality of Professional Development and the District Capacity items

initially contained an error; the “Neither disagree nor agree” option offered only “Agree.”

Because it is not possible to know if the participants who saw this item were influenced by this

erroneous response option, all participants who viewed the incorrect scale were removed from

that section of the analysis.

Analysis of the participant survey data focused on frequency of responses. Data presented from

this survey in the following sections either show proportions of respondents who agreed or

strongly agreed with each item, or they show the frequency of all participant responses,

depending on the scale used.

LEA Professional Development Coordinator Interviews

Principals at each sample school were asked to identify the person in their LEA who coordinated

professional development training and assistance for their school. The Professional Development


March 2013


Coordinators were asked to participate in a 30-minute interview to discuss their LEAs’ capacity

to provide high-quality professional development. The interview protocol (Appendix B)

contained questions specific to strategies used in selecting appropriate professional development

for teachers and staff, integration of RttT priority areas into professional development training,

and the use of state-provided resources. Interviews were conducted between October 2011 and

March 2012.

Principal Interviews

Principals from each sample school participated in two 30-minute semi-structured interviews in

the fall and spring semesters, during which they discussed their experience with professional

development planning and implementation. The interview protocol (Appendix C) contained

questions specific to strategies used in selecting appropriate professional development for

teachers and staff, integration of RttT priority areas into professional development training, and

the use of state-provided resources.

Teacher Focus Groups

Six to eight core content teachers (in English, Math, Science, and Social Studies) from each

school were selected to participate in a 45- to 60-minute focus group. Teachers from each school

were assembled onsite and engaged in a discussion facilitated by an Evaluation Team member.

The focus group protocol (Appendix D) was designed to gather teachers’ opinions of the

availability and use of professional development resources at their school, as well as of their

overall impressions of the transition to the new Common Core and Essential Standards.

The Evaluation Team collected over 45 hours of interview and focus group data. After each

audio recording was transcribed, Atlas.ti software was used to facilitate qualitative analysis of

the data. An a priori coding scheme was developed based on the evaluation questions outlined in

the introduction of this report. The coding scheme was refined and expanded as the team

interacted with the data. Results from the analysis contributed to the baseline outcomes in this

report.

Limitations

Due to changing schedules and available resources on the RttT Professional Development

Evaluation Team, a few data collection strategies did not reach their full potential.

CLASS Observation Protocol: Evaluators were able to observe only a small proportion of the

teachers in the schools in our longitudinal study, and only for a short time—four teachers in a

school for two hours at a time, twice a year. The intent is not to rate teachers but to use this

valid and reliable classroom observation tool to systematically record observed instructional

interactions between teacher and students. For the purposes of this study, these data will be

used to describe teachers’ transition to the new state content standards (Common Core and

Essential Standards), as evidenced by effective instructional practices required for the

teaching to the new standards.

Teacher Professional Development Survey: When transferring the survey to the online survey

portal, a clerical error resulted in two “agree” response options for the items about Quality of


March 2013


Professional Development and LEA Capacity. Because it is not possible to know if

participants were influenced by this erroneous response option, all participants who viewed

the incorrect scale for those constructs were removed from analysis.

Specific strategies have been identified to address these limitations for the remainder of the

evaluation and are outlined in the final section of this report, Next Steps for the Professional

Development Evaluation.


March 2013


Results

This report provides a baseline (first-year) assessment of the State’s progress toward updating

the education workforce in North Carolina, as measured by the status of intermediate outcomes

related to that goal (Figure 3). These outcomes are expected to result from educator participation

in statewide face-to-face RttT professional development events designed to impact (a) local

capacity to provide high-quality professional development, (b) shifts in local organizational

conditions to support RttT priorities, and (c) changes to instructional practice. That is, this report

provides baseline data for addressing Evaluation Question 3— To what extent did RttT

professional development efforts successfully update the North Carolina education workforce?—

and related sub-questions.

Figure 3. Logic Model of Local-Level Outcomes for Statewide RttT Professional Development

*RttT priorities for North Carolina include: successfully transitioning to the new standards, implementation of

formative and summative assessments, use of data to support instruction, and effective utilization of the new North

Carolina Educator Evaluation System (NCEES).


of organizational change, including capacity to provide high-quality local professional

development aligned to RttT priorities?

In order to assess progress toward meeting intermediate outcomes related to updating the North

Carolina education workforce at this point in the RttT grant, qualitative and quantitative data

were collected to: (1) describe whether local capacity has been built to plan, access, and

implement high-quality professional development (Figure 4, following page); (2) provide

baseline information about the extent to which there have been organizational shifts at the school

and LEA level related to the RttT priorities; and (3) provide baseline information about the

quality of the professional development being provided.


March 2013


Figure 4. Logic Model of Outcomes for Statewide RttT Professional Development—Local

Capacity Focus




Planning, Accessing, and Implementing Local Professional Development

Planning and Accessing Local Professional Development. Focus group and interview data

revealed that, for most schools, school-level professional development was informed by what

was being offered at the state level and typically was designed around resources available at the

LEA level. When asked about professional development training around specific RttT priority

areas in particular, some coordinators and principals noted a strong reliance on state and LEA

guidance (e.g.: “We go with whatever is provided by our LEA;” “I think our professional

development is through what is offered by the state”). In the few cases where local-level

leadership were more directly involved in the planning process, Professional Development

Coordinators and principals highlighted cross-LEA sessions where professional development

strategies were exchanged among LEAs in their region. One Coordinator’s comments

summarized the appreciation expressed by local leadership for the chance to engage in the

planning process:

When you’re working within a district, you’re isolated within that district and you don’t

know what else is going on around you. It was good to hear what smaller districts could

do, and what some of the larger districts comparable to [our] county could be doing.

Since we are all going toward the new assessments and new curriculum, it’s good to hear


March 2013


what others are doing to prepare their teachers. Then we can decide within our county

cohort what things we think might work, if we need to go back in and redesign our plan,

or move forward with what we think our best practice is.

Some schools also relied on in-house experts or outside vendors (e.g., Reading 3D or PD 360) to

plan supplemental professional development training for their teachers.

Qualitative data from the sample schools detailed the ways in which information about

professional development availability typically was disseminated by LEA leadership to teachers

and principals at the school level. In line with the state-endorsed “train the trainer” model, LEA-

level Professional Development Coordinators indicated that they were primarily responsible for

identifying and disseminating the most relevant professional development resources and making

them available to principals and teachers in their LEAs. Most LEA Coordinators mentioned

using tools like wikis, website postings, and staff development newsletters and calendars to

condense professional development resources and make them more easily accessible to teachers

and principals. Other methods of increasing awareness of professional development

opportunities included teachers and administrators sharing information at faculty meetings and in

subject-area or grade-level Professional Learning Communities (PLCs).

Despite the multiple methods of information dissemination described by LEA leaders, data

revealed that some principals and teachers still remained uninformed about NCDPI-supported

professional development opportunities. For example, when focus group moderators asked

teachers whether they were aware of any online professional development opportunities offered

by NCDPI, teachers from four schools were unaware of any of the online modules being offered,

and teachers from four other schools indicated that they were aware of or had accessed the NC

FALCON modules developed prior to RttT but were not aware of the new online modules

released in the summer and fall of 2011. This lack of awareness was likely due to either a LEA

Professional Development Coordinator’s decision to postpone the rollout of the modules,

unresolved decisions about what modules or module components to use, or, as in the case of two

LEAs, a decision not to make the modules mandatory because of duplication of prior local

efforts or concerns over their quality.

While most schools were able to locate and access some professional development resources,

more evidence is needed to fully assess changes in local professional development training

patterns. While there are indications that, at the state level, NCDPI leadership has recognized the

need to create a better structure to describe how to connect state-implemented professional

development to professional development training at the LEA and school levels, a need for

specific communication strategies for professional development at the local level persists, as

evidenced by teachers’ and principals’ lack of awareness of professional development resources,

despite LEA Professional Development Coordinators’ express role in disseminating state-

provided professional development resources. Differences in professional development needs

across schools and the availability of resources impact North Carolina educators’ ability to

translate state-level professional development resources into local-level training. This baseline

data will be used as a point of comparison for future evaluations of LEA and school ability to

locate and access professional development resources.


March 2013


Implementing Local Professional Development. The Evaluation Team’s approach to examining

LEA and school preparedness to implement professional development was two-fold. First, data

from the statewide LEA RttT Professional Development Coordinator Survey was used to

examine the extent to which LEA leaders believed they had the capacity to implement high-

quality professional development. Second, qualitative data from principals and teachers in our

sample schools provided detailed information about the process by which LEA and school

leadership selected, planned, and implemented professional development at the local level.

Table 8 presents results from the RttT LEA Professional Development Coordinator Survey on

perceptions of LEA leadership capacity to develop professional development programs that suit

the needs of their schools and LEAs.

Table 8. Perceptions of LEA PD Coordinators about LEA Capacity to Implement Professional

Development

In my district, leaders…

Percentage of

Respondents

who Strongly

Agree/Agree

(n = 144-150)

Have knowledge about high-quality professional development defined by research

and national and state standards 83%

Have the skills to plan and design high-quality professional development 83%

Have the skills to implement high-quality professional development 87%

Have the skills to select high-quality professional development 92%

Have positive attitudes about high-quality professional development 94%

Have a district-wide commitment to high-quality professional development 91%

Support Communities of Practice around high-quality professional development 81%

Provide opportunities for networking and support (both online and offline) in high-

quality professional development 77%

Have a district-level strategic plan for professional development in place 86%

Align school-level professional development plans to district plans 80%

Distribute responsibilities for leadership for professional development among

multiple school and district leaders 86%

Use data from teacher performance evaluations to create individual professional

development plans for teachers 68%

Aggregate data from teacher performance evaluations and use data to identify

school/district professional development needs (Teacher Professional Development

Survey: Use data from teacher evaluations to identify school/district professional

development needs)

64%

Use survey data to select, plan, and design professional development 78%

Use summative student data to select, plan, and design professional development 79%

Use formative student data to select, plan, and design professional development 67%

Source: RttT LEA Professional Development Coordinator Survey


March 2013


Survey results revealed that statewide, a large majority of Professional Development

Coordinators agreed or strongly agreed that their LEA leadership had the capacity to implement

and plan various aspects of professional development. For instance, 91% of Professional

Development Coordinators agreed or strongly agreed that their LEA leaders had a commitment

to implementing high-quality professional development, and 86% of Professional Development

Coordinators believed that their LEAs had a strategic plan in place for professional development.

Results were slightly less favorable regarding LEA leaders’ use of data to inform professional

development decisions, with 79% of coordinators agreeing that LEA leaders used summative

student data to select, plan, and design professional development, 67% agreeing that leaders used

formative student data for the same purpose, and only 64% agreeing that LEA leaders used data

from teacher performance evaluation results to identify professional development needs.

The design for professional development training implementation sometimes was a group effort

at the LEA level (for example, collaboration among principals from across an LEA), as

explained by this focus group participant:

This particular year, how we are rolling out the Common Core, we brought in a group of

principals, elementary level, middle school level, and high school level, to look at how

they thought we should roll this out, and who should be trained, and who should be the

ones taking the information back to the schools. We are very large. And so there’s no way

that we’re going to have the face-to-face contact with each teacher within the district. So

we had to develop a plan to use the trainer model, so to speak, in order to at least get the

information out, so that they can begin the teaching of the Common Core.

At the school level, professional development training was designed primarily for delivery

within PLCs or staff meetings. Some schools provided supplemental training resources through

online tools (e.g., Live Binder or Moodle) as additional support for teachers in their transition to

the new state standards. Generally, teachers felt that PLCs were designed to provide an

opportunity to gather with other teachers in their content areas or grade levels to review

professional development resources and walk through any changes in curriculum related to the

new state standards that would affect their instructional practice. For many teachers, PLCs or

staff meetings were their first entrée into professional development training focused on RttT. The

following comments reflect teacher and principal feedback regarding the usefulness of designing

PLCs around professional development goals:

I think obviously the professional learning communities, it’s really helped us to grow

those and it has also enabled us to really utilize that blended approach where we can use

some online, some face-to-face, and we’re using Moodle.

We’re asking teachers to use these [additional] tools to learn the standards themselves.

We’re using the online environment to actually use these modules that DPI set up and

then take them a step further and use the Moodle environment to hold the PLC

conversations. So once they view the module, then we have them go and do online

discussions within their PLC, and so each PLC is doing something a little differently.

Implementation was not limited to 2011-12 school year professional development. Many LEAs

provided summer pre-training opportunities designed to give teachers early chances to explore


March 2013


new Common Core materials and find ways to apply them to their specific grade levels. Most

teachers in the focus groups noted that their experience in summer training was helpful in

preparing them for the use of state-implemented strategies related to Common Core. For a small

proportion of LEAs and schools, implementation of professional development around the new

state standards began one or two years before the 2011-12 school year. One Professional

Development Coordinator commended her LEA for extending professional development

strategies around the Common Core and Essential Standards from the previous year:

Our county is trying to be proactive and they made sure at the end of the last school year,

the 2011-11 school year, that we had the actual Common Core curriculum in our hands,

and then over the summer and through this year, they’re making sure we’re unpacking

the standards; looking at them, understanding for each grade level: Do you know what is

being transitioned out of your grade? Do you know what’s being transitioned into your

grade? I know that there are a lot of counties that are surrounding us that are not doing

those things to prepare. We’re already beginning to teach those things and starting to

close the gap this year so that it won’t be as hard of a transition for the teachers next

year.

Alignment with RttT Priorities

Items from the LEA Professional Development Coordinator Survey and Professional

Development Teacher Survey specifically asked about teachers’ participation in professional

development aligned to RttT priorities (Table 9). Only about half of LEA Professional

Development Coordinators from across the state and teachers in the sample schools agreed or

strongly agreed that teachers built their knowledge and skills for revised state assessments (51%

and 52%, respectively) and teaching diverse student populations (53% and 59%, respectively).

Perceptions of LEA Professional Development Coordinators statewide and teachers in our

representative sample differed in other areas, most particularly about whether teachers built their

knowledge and skills for revised state standards (91% and 64%, respectively) and whether they

deepened their content knowledge (81% and 58%, respectively).

Table 9. Perceptions of Participation in Professional Development on RttT Priorities

To what extent do you feel that teacher knowledge and

skills have been enhanced in each of the following areas as a

result of participation in professional development?

Percentage of Respondents who

Strongly Agree/Agree

PD Coordinators

(n = 146-148)

Teachers

(n = 864-890)

Revised state standards 91% 64%

Instructional materials 73% 58%

Approaches to formative assessment 79% 64%

Use of technology 67% 70%

Strategies for teaching diverse student populations 53% 59%

Deepening content knowledge 81% 58%

Leadership development 71% 53%

Revised state assessments 51% 52%

Revised NC Teacher Evaluation Process 81% 63%

Sources: RttT LEA Professional Development Coordinator Survey, RttT Professional Development Teacher Survey


March 2013


When asked to rate their level of agreement with statements about the alignment of their LEA’s

professional development training and RttT priorities, most coordinators (>85%) agreed that their

LEAs had plans, strategies, resources, and systems in place to support RttT Professional

Development (Table 10). Fully 92% of coordinators reported that LEA-provided RttT professional

development was aligned with and built upon existing professional development initiatives.

Table 10. Professional Development Coordinators’ Perceptions of LEA Capacity to Support RttT

Professional Development

To what extent do you agree/disagree with the following?

Percentage of

Respondents who

Strongly Agree/Agree

(n = 139-144)

Our Race to the Top professional development is aligned with and builds on existing

professional development initiatives in the district. 92%

Our district has . . .

a strategic plan for the deployment of Race to the Top professional development. 92%

an implementation timeline in place for Race to the Top professional development. 90%

a process in place for how we will communicate Race to the Top professional development

initiatives and opportunities to all district staff. 92%

ensured that Race to the Top professional development activities are supported with up-to-

date board-approved policies and procedures. 85%

designated key personnel who are responsible for various components of our Race to the Top

professional development plan. 93%

a system for monitoring and evaluating our Race to the Top professional development plan. 79% communicated with our local testing coordinator to ensure all staff has NC Education login

capabilities. 88%

a comprehensive plan to coordinate the integration of face-to-face and online professional

development. 78%

utilized data on staff’s technology proficiency when planning Race to the Top professional

development. 64%

ensured all staff has equitable access to technology for accessing Race to the Top

professional development resources. 85%

determined that all online resources related to Race to the Top are accessible in the LEA. 89%

specific strategies for collaboration as we roll out Race to the Top professional development. 91%

provided educators an online space (e.g., wiki, website, Moodle, etc.) for sharing resources,

experiences, and/or information to related to RttT. 77%

collaborated with district technology staff to determine the extent of online communication

and collaboration tools available for PD. 72%

designated key personnel to be responsible for monitoring and recording the progress of staff

in completing online and face-to-face Race to the Top PD. 81%

developed a plan for implementing PLCs (online and/or face-to-face) related to Race to the Top. 73% designated key personnel responsible for scheduling and monitoring PLCs related to Race to

the Top. 70%

a plan for how to use the DPI PD Leads in each region for support in our Race to the Top

professional development. 60%

a long-range plan in place for Race to the Top professional development for the next three years. 71% Our district’s Race to the Top professional development . . .

goals are aligned with state initiatives. 94%

team can articulate their role in our Race to the Top professional development plan. 85%

differentiates to meet participants’ needs with additional resources and training. 75%



March 2013


However, coordinators highlighted two areas in which they thought LEAs were less effective in

implementing RttT professional development. Only 64% of coordinators agreed that their LEAs

utilized data about staff’s technology proficiency when planning RttT professional development.

In addition, only 60% agreed that their LEAs have plans for how to use the NCDPI Professional

Development Leads in each region for support in their RttT professional development. Similar to

some of the results reported above regarding perceptions of LEA capacity to provide appropriate

professional development, these results also suggest a lack of capacity to use data to guide the

selection, design, and implementation of high-quality professional development.

Quality of Local Professional Development

Data from the RttT Omnibus Teacher Survey provides insight into North Carolina teachers’

general perceptions of the overall quality of professional development they experienced in 2011

and 2012 (Table 11). About 70% of teachers agreed or strongly agreed that professional

development experiences were both sustained and coherent in the 2010-11 and 2011-12 school

years. However, fewer (63%) agreed or strongly agreed that they had enough time to think about,

try, and evaluate new ideas in 2010-11, and that number fell further (to only 56%) in 2011-12.

Table 11. Teacher Perceptions of Overall Quality of Professional Development

Overall, my professional development experiences this year have...

2011

Strongly

Agree/Agree

(n = 1,709)

2012

Strongly

Agree/Agree

(n = 10,011)

Been sustained, rather than short-term. 70% 70%

Been coherently focused, rather than unrelated. 70% 69%

Included enough time to think carefully about, try, and evaluate new ideas. 63% 56%

Source: RttT Omnibus Teacher Survey

Both the RttT Omnibus Teacher Survey and LEA Professional Development Survey results

demonstrate that, although a majority of respondents agreed that their experiences were positive,

teachers generally gave lower ratings for their experiences with LEA-provided professional

development than did LEA Professional Development Coordinators (Tables 11 and 12).

In their assessment of the quality of LEA-provided professional development, 93% of the

Professional Development Coordinators agreed that teachers were provided opportunities to

work productively with colleagues in their school, compared to only 70% of teachers. Similarly,

66% of Coordinators but only 46% of teachers agreed that teachers were provided the

opportunity to work with teachers from other schools, and 81% of Coordinators but only 63% of

teaches thought most of what teachers learned in professional development addressed the needs

of the students in the classroom (Table 12, following page).


March 2013


Table 12. Professional Development Coordinators’ Perceptions of LEA-Level Professional

Development Alignment

To what degree do you agree with the following statements about

the PD experiences provided in your district this year?

Strongly

Agree/Agree

(n = 152-154)a

Strongly

Agree/Agree

(n = 10,011)b

Teachers are provided opportunities to work productively with

colleagues in their school 93% 70%

Teachers are provided opportunities to work productively with

teachers from other schools 66% 46%

Most of what teachers learn in professional development addresses the

needs of the students in their classroom 81% 63%

Sources: a RttT LEA Professional Development Coordinator Survey;

b RttT Omnibus Teacher Survey

Data gathered in interviews and focus groups that was related to the quality of local professional

development was somewhat more encouraging, highlighting principals’ and teachers’ positive

experiences with local coaches and specialists, as well as their positive impressions of the value

of PLCs. However, mirroring survey results, some respondents focused on the challenge of

finding time to participate in and then implement what they learned in local-level professional

development.

Many of the teachers and principals who discussed their experiences with local-level

professional development experts focused on the benefits of having qualified in-house

facilitators (e.g., lead teachers, media specialists, technology coaches) and other LEA-level

trainers available to provide professional development on-site in their schools:

The district this year has hired a secondary literacy lead teacher who is providing

professional development for us on summarizing and just more awareness of literacy in

the content area.

[The Medial Specialist] provide[d] trainings in the afternoon! It might [have been]

something that the teachers came up and said, “We want some more information on

Promethean Boards, how to use them, and just put it in our lesson plan.” He provided a

quick training in the afternoon for like an hour.

With regard to PLCs, some teachers mentioned how valuable it was to their instructional practice

to engage in group discussion about the professional development materials. In a few cases,

participants gave examples of how formal professional development training was brought more

into their informal conversations during PLCs, indicating that professional development was

being integrated into teachers’ ongoing practices. In the passages below, one teacher expresses

the importance of PLCs, and a principal recounts an occasion in which material from a PLC

training became part of his teachers’ collaborative efforts.

I like the PLCs in that we have the data that we can review to see what areas we need to

focus on to enhance the learning that’s going on in the classrooms. So you’re not

isolated. You’re working, of course, in a group and you’re helping each other.


March 2013


I find another plus, too, with [teachers], they have some experience in lower grades and

that alignment in going to the different grade levels and passing this information on. I see

them talk in the hallway discussing this information of how I might react to it, or after

our PLCs, how it would translate into 3rd

grade or 4th

grade, whichever grade on their

part.

We feel like we’ve been transitioning for a couple of years now. We did have a workshop

back in the summer where we could prepare our pacing guides and even working

towards these transitions from the last couple of years [but] it is more beneficial

whenever we can meet as a whole math department in those types of settings … the PLCs

definitely help zero in on our subject.

As noted above, several focus group participants also mentioned that they felt inhibited in their

ability to engage in professional development, often citing lack of time. Teachers in particular

mentioned lack of time as a major impediment to application of the professional development

training they had received:

Time for me will be the biggest challenge [to implementation]. It’s just we’re given these

new standards [dealing with] literacy, the Common Core … and I just feel like I would be

capable of planning really good lessons to fulfill those new standards but just having the

time to do that would be my biggest concern right now.

Summary of Findings for Evaluation Question 3A

Overall, teachers and Professional Development Coordinators were confident in their LEA

leaders’ ability to provide their schools with high-quality professional development. Although

selecting, planning, and designing processes varied across schools and LEAs, results generally

indicated that educators were pleased with the quality, alignment, and initial impact of local

professional development training. Even though some schools have attempted to assuage the

time strain for teachers by designing professional development trainings that take place the year

or summer before the academic year, educators in the sample consistently referenced a lack of

time as a central implementation issue.


March 2013


Evaluation Question 3.B.: How did school culture/organizational conditions change to

support RttT priorities?

Figure 5. Logic Model of Local Outcomes for Statewide RttT Professional Development –

Organizational Focus




A combination of LEA Professional Development Coordinator Surveys, Teacher Professional

Development Surveys, and RttT Omnibus Teacher Surveys provided data for estimating the

early status of organizational shifts related to RttT priorities for the 2011-12 school year.

Changes in Conditions related to Transition to the New State Standards

LEA Professional Development Coordinators indicated that, for the 2011-12 school year,

planning and strategy at the LEA level for providing professional development that supported the

roll-out of the Common Core State Standards and North Carolina Essential Standards varied

considerably (Table 13, following page). Nearly all LEA Professional Development

Coordinators surveyed (92%-94%) agreed that their LEAs had a process in place for

communicating the revised state standards and upcoming professional development opportunities

and had specific strategies in place to support LEA collaboration as they roll out professional

development on revised state standards, and almost all of the coordinators (86%-91%) indicated

that their LEAs had a specific plan for using NCDPI’s Unpacking and Crosswalk Documents

during revised state standards professional development. However, only about one half to two-


March 2013


thirds of respondents indicated that their LEAs had specific strategies for integrating Exceptional

Children (66%), Academically and Intellectually Gifted (58%), and English Language

Development (56%) components into professional development on the revised state standards.

Table 13. Professional Development Coordinators’ Perceptions of LEA-Level Professional

Development on the Revised State Standards

To what extent do you agree/disagree with the following?

Strongly

Agree/Agree

(n = 142-145)

Processes

Our district has a process in place for how we will communicate revised (Common

Core and Essential Standards) State Standards professional development

opportunities.

94%

Our district has specific strategies for collaboration as we roll out revised (Common

Core and Essential Standards) State Standards professional development. 92%

Integration

Our district has specific strategies for integrating AIG into revised (Common Core

and Essential Standards) State Standards professional development. 58%

Our district has specific strategies for integrating EC into revised (Common Core and

Essential Standards) State Standards professional development. 66%

Our district has specific strategies for integrating ELD standards into revised

(Common Core and Essential Standards) State Standards professional development. 56%

Our district has specific strategies for integrating Information and Technology

standards into revised (Common Core and Essential Standards) State Standards

professional development.

72%

Our district has specific strategies for integrating 21st century skills into revised


Use of Related State Resources

Our district has a specific plan for using the Crosswalk documents during revised


Our district has a specific plan for using the Unpacking documents during revised



Items on the RttT Omnibus Teacher Survey asked about teachers’ awareness of and expectations

for the revised State Standards (Table 14, following page). Results showed that nearly all (91-

95%) teachers felt strongly about their own awareness and knowledge linked to the state

standards but were less sure about whether school leaders (69-76%) or other teachers (54-80%)

in their school really understood how to implement the new standards effectively.


March 2013


Table 14. Teachers’ Perceptions of Organization-Level Attention to the Revised State Standards

Items

Strongly

Agree/Agree

(n = 10,011)

I fully understand the content knowledge and skills that are needed to master the

subject(s) I teach. 95%

I have the knowledge and skills I need to help my students accomplish what is

expected of them in the state standards. 95%

I feel well-prepared to help students accomplish what is expected of them in the state

standards. 91%

Leaders in my school…

Discuss the state standards with teachers. 76%

Provide opportunities for teachers to discuss understandings the state standards

across grades and content areas. 69%

Teachers in my school…

Coordinate curriculum across grade levels according to the state standards. 64%

Are clear about learning goals based on the state standards. 80%

Have a clear understanding of how the state standards fit together across grade levels. 65%

Focus more on teaching the state standards than “teaching to the test.” 54%

Work together on developing/revising instructional units around state standards. 63%

Are prepared to help students meet the state standards. 71%

Source: 2012 RttT Omnibus Teacher Survey

Changes in Conditions related to Data-based Decision Making and Revised Teacher Evaluation

Process

In addition to helping educators transition to the new Common Core and Essential Standards,

RttT professional development also is supposed to address two other RttT priority areas: the use

of data for decision-making and the transition to a revised and expanded teacher evaluation

process. For this report, early indirect evidence for the impact of professional development on

both of these areas was drawn from two years of teacher responses to related items on the

Omnibus Survey.

The Omnibus Survey data revealed a small but consistent trend in teacher perceptions of school-

wide practices and attitudes related to both data-based decision-making and the teacher

evaluation process. There were slight decreases in teacher agreement between 2010-11 and

2011-12 responses across all items, though these changes could be accounted for by a sizeable

increase in responses rates (from 1,709 in 2010-11 to 10,011 in 2011-12). The Evaluation Team

will continue to track responses in future years to determine whether the patterns persist.

For example (Table 15, following page), in 2012: 86% of teachers agreed or strongly agreed that

teachers collect and use data to improve their own teaching, down from 88% in 2011; 71%

agreed or strongly agreed that they meet regularly with others teachers to review student data

together (down from 73%); and only 61% agreed or strongly agreed that their whole school

examines gaps in achievement of students by grade level (down from 64%).


March 2013


Table 15. Teachers’ Perceptions of Organization-Level Data-Driven Instructional Activities

At my school...

2011

Strongly

Agree/Agree

(n = 1,709)

2012

Strongly

Agree/Agree

(n = 10,011)

Teachers collect and use data to improve their teaching. 88% 86%

We have made changes designed to better meet the needs of its diverse

student body. 83% 72%

Teachers are engaged in systematic analysis of student performance data. 84% 71%

Assessment of student performance leads to changes in our school’s

curriculum that are consistent with state standards. 76% 74%

Useful information to make informed decisions is readily available to

teachers (e.g., about student performance, resources, community

satisfaction)

70% 66%

We use assessment data to evaluate teachers’ instructional practices. 70% 67%

The whole school examines gaps in the achievement of students by

grade level. 64% 61%

Teachers meet regularly to review student performance in order to

adjust their practices. 73% 71%

We use a variety of assessment strategies to measure student progress. 82% 79%


Similarly (Table 16), in 2012, 71% of teachers agreed or strongly agreed that the evaluation

process encourages teachers to reflect on their practice, down from 76% the previous year. Only

64% agreed or strongly agreed that the evaluation process encourages professional growth (down

from 70%), and just 58% that the teacher evaluation process was fair (down from 64%). Also,

only 51% (down from 54%) agreed or strongly agreed they were satisfied with the teacher

evaluation process.

Table 16. Teachers’ Perceptions of Organization-Level Teacher Evaluation Activities

Item

2011

Strongly

Agree/Agree

(n = 1,709)

2012

Strongly

Agree/Agree

(n = 10,011)

At my school . . .

The evaluation process encourages teachers to reflect on their

instructional practice. 76% 71%

Teachers use feedback from the teacher evaluation system to

improve their teaching. 71% 69%

Teacher evaluation is fair. 64% 58%

The criteria on which I am evaluated are clear. 63% 63%

The teacher evaluation process encourages professional growth. 70% 64%

This school year, overall…

I am satisfied with the teacher evaluation process 54% 51%



March 2013


Evaluation Question 3.C.: To what extent did teachers improve classroom practice?

Figure 6. Logic Model of Local Outcomes for Statewide RttT Professional Development –

Classroom Practice Focus




The content-specific items in the Teacher Professional Development Survey assessed the extent

to which teachers believed they used their instructional time to allow students opportunities to

engage in learning strategies aligned with the Common Core in English and Math, and with

Essential Standards for Social Studies and Science. Across each content area, results showed that

teachers felt students spent the least amount of their daily instructional time on the highest-level

cognitive processing skills and tasks. Overall, Math teachers reported spending the greatest

proportion of their daily instructional time on an array of the instructional strategies related to the

new state standards, followed by English Language Arts, Science, and Social Studies teachers

(Tables 17-20). These data provide baseline information about current learning activities against

which the Evaluation Team will compare data in subsequent years of RttT.

With regard to Common Core English Language Arts-related learning strategies (Table 17,

following page), the largest percentage of teachers estimated that students spent time almost

daily on listening skills (76%) and on general vocabulary (66%). The lowest percentages of

teachers reported that students spent daily instructional time defending well-developed claims

(28%), challenging their peers with relevant follow-up questions (30%), and analyzing and

synthesizing sources (33%).


March 2013


Table 17. Teacher Estimations of Time Spent Engaged in English Language Arts Activities

How much of the ELA instructional time do

students use to engage in the following tasks? n Never

A Few

Times a

Year

Once or

Twice a

Month

Once or

Twice a

Week

Almost

Daily

Read increasingly complex texts with increasing

independence 235 1% 1% 6% 32% 60%

Analyze and synthesize sources 231 3% 3% 10% 50% 33%

Present careful analysis, well-defended claims,

and clear information 231 3% 6% 18% 45% 28%

Gain listening skills 232 1% 0% 4% 18% 76%

Respond to and challenge their peers with relevant

follow-up questions and evidence 234 3% 2% 18% 47% 30%

Acquire knowledge of general academic

vocabulary 232 0% 0% 4% 29% 66%

Draw evidence from texts to support their response 234 0% 0% 4% 30% 65%

Read deeply to gain knowledge from texts 234 1% 2% 5% 32% 60%

Source: RttT Professional Development Teacher Survey

Among Math teachers (Table 18), 64% to 80% said their students engaged in five of nine

Common Core Math-related learning strategies almost daily. In particular, the highest percentage

of teachers reported that on an almost daily basis, students made sense of problems (80%),

persevered in solving problems (72%), and used appropriate tools strategically (67%). Least

frequently practiced by students almost daily were construction of viable arguments (23%) and

critiques of the reasoning of others (24%).

Table 18. Teacher Estimations of Time Spent Engaged in Mathematics Activities

How much of the Mathematics instructional time do


A Few

Times a

Year

Once

or

Twice a

Month

Once or

Twice a

Week

Almost

Daily

Make sense of problems 224 0% 0% 2% 18% 80%

Persevere in solving problems 223 0% 1% 3% 25% 72%

Reason abstractly 221 1% 1% 9% 49% 40%

Reason quantitatively 222 0% 1% 8% 36% 55%

Construct viable arguments 221 3% 4% 15% 55% 23%

Critique the reasoning of others 221 4% 7% 22% 43% 24%

Model with mathematics 223 0% 1% 5% 30% 64%

Use appropriate tools strategically 223 1% 1% 5% 26% 67%

Attend to precision 222 1% 2% 9% 25% 64%



March 2013


Science teachers reported (Table 19) that their students utilized most of the learning strategies

related to new Essential Standards for Science—including conducting investigations, testing

predictions, measuring, and keeping records—at least once or twice a month. On a daily basis,

Science teachers reported that their students participated in hands-on activities, completed

activities with a real-world context, and used tools to gather data more often than they tested

predictions or kept accurate records of their investigations.

Table 19. Teacher Estimations of Time Spent Engaged in Science Activities

How much of the Science instructional time do


A Few

Times a

Year

Once or

Twice a

Month

Once or

Twice a

Week

Almost

Daily

Develop problem-solving skills through

investigations 174 1% 2% 25% 57% 16%

Make predictions that can be tested 173 2% 5% 32% 51% 10%

Use tools to gather data (e.g., calculators, computers,

graduated cylinders, scales, and meter sticks) 173 1% 6% 33% 41% 19%

Measure with accuracy 172 1% 10% 46% 35% 8%

Conduct multiple trials of an investigation to test a

prediction 173 6% 18% 38% 33% 5%

Keep accurate records of investigation trials 172 6% 12% 40% 35% 6%

Create reasonable explanations of results of an

experiment or investigation 172 3% 8% 38% 42% 9%

Choose the most appropriate mechanism to express

results (e.g., scientific language, drawings, models,

charts, or graphs)

171 6% 8% 39% 37% 11%

Utilize appropriate safety procedures when

conducting scientific investigations 172 2% 9% 30% 41% 17%

Participate in hands-on-activities 172 1% 1% 23% 43% 32%

Complete activities with a real-world context 172 1% 2% 24% 49% 24%

Engage in technological design investigations 170 9% 12% 39% 32% 7%

Conduct scientific investigations 170 2% 11% 38% 42% 7%


Finally, like Science teachers, Social Studies teachers reported allocating much of their time to

engaging students in all of the Essential Standards-related tasks (Table 20, following page). A

notable proportion of Social Studies teachers felt that their students spent time almost daily on

recognizing and appreciating contributions of diverse cultural groups (30%), demonstrating

chronological thinking (29%), and analyzing cause-and-effect relationships (28%). Least likely

to be done daily were constructing historical narratives (5%) and analyzing primary documents

(11%).


March 2013


Table 20. Teacher Estimations of Time Spent Engaged in Social Studies Activities

How much of the Social Studies/History

instructional time do students use to engage in the

following tasks? n Never

A Few

Times

a Year

Once or

Twice a

Month

Once or

Twice a

Week

Almost

Daily

Demonstrate chronological thinking 159 1% 3% 18% 48% 29%

Explore changes in communities and regions over time 160 2% 5% 28% 42% 23%

Recognize and appreciate the contributions of

diverse cultural groups 162 1% 3% 21% 45% 30%

Explain why people can describe the same event

differently 161 4% 3% 32% 41% 19%

Draw connections between contemporary issues

and their historical origins 160 4% 6% 25% 43% 23%

Transfer understanding from the state to the

national level 158 7% 10% 36% 34% 13%

Articulate the implications of increased global

interactions 159 4% 12% 34% 36% 13%

Recognize and interpret the “lessons of history” 159 6% 9% 31% 40% 15%

Identify pivotal moments in world history that

shaped the development of contemporary societies 160 4% 11% 36% 33% 16%

Use geography to understand current global

conditions 160 6% 12% 31% 39% 13%

Identify patterns of continuity and change 159 5% 9% 28% 41% 16%

Articulate the roles of the government 159 3% 17% 28% 33% 19%

Use maps, charts, and graphs, and other

geographic tools 162 1% 6% 23% 46% 24%

Compare multiple perspectives and interpretations

of the same issue, time period, etc. 160 5% 16% 28% 36% 14%

Analyze primary documents and other artifacts 160 9% 15% 37% 29% 11%

Construct historical narratives 160 14% 19% 41% 21% 5%

Analyze cause-and-effect relationships 159 3% 4% 21% 45% 28%


Classroom observations using the CLASS Observation tool yielded a baseline understanding of

how teacher-student interactions in the classroom reflect specific instructional practices outlined

in the new state standards. The Evaluation Team identified Regard for Student Perspectives

(RSP), Content Understanding (CU), Analysis and Problem Solving (APS), Quality Feedback

(QF), and Instructional Dialogue (ID) as the dimensions of the CLASS Observation tool that

most closely align with the implementation of the new state standards (Table 3, above). Through

the lenses of these dimensions, the Evaluation Team observed teacher-student interactions as

teachers delivered lessons in English Language Arts, Math, Science, and Social Studies classes.

These classroom visits revealed that, on average, teacher-student interactions demonstrated some

of the key characteristics associated with the new state standards.

Scores in CLASS are classified as being either Low (1 or 2), Midrange (3, 4, or 5), or High (6 or

7). On average, classrooms across subject areas in the sample schools received midrange ratings

across each of the CLASS dimensions. Midrange CLASS scores generally indicate an average

display of skills or behaviors associated with each CLASS dimension. Classrooms observed for

this evaluation typically scored highest on the dimensions of Content Understanding (mean =


March 2013


3.98, SD = 1.42) and Regard for Student Perspective (mean = 3.94, SD = 1.38), indicating that

the focus of most classes in the sample tended to fluctuate between meaningful discussion with

explanation of broad organizing ideas and presentation of discrete bits of topically-related

information. The scores also suggest that in sample core content classrooms, the teacher

generally provided structure for the class but remained flexible and occasionally integrated

student perspectives or experiences into their lessons. Evidence of consistent, quality application

of strategies in the dimensions of Quality Feedback (mean = 3.75, SD = 1.35) and Instructional

Dialogue (mean = 3.38, SD = 1.48) was weaker, but still within the “Mid” range. These results

suggest that, although teachers in our sample sometimes used facilitation strategies (e.g., open-

ended questions, repetition/extension, active listening) to encourage more elaborated content-

based dialogue among students, observed student interactions were often perfunctory. Observed

classrooms were least often engaged in Analysis and Problem Solving Skills (mean = 2.80, SD =

1.35), which means more time might need to be spent in the classroom on higher-order thinking

skills and application of student knowledge and skills within familiar contexts. On average, in

content area classrooms across the sample, teachers sometimes scaffolded discussion in the

classroom and made attempts to integrate student feedback in different parts of their lessons.

Table 21 shows the distribution of CLASS ratings across content area classrooms (ELA, Math,

Science, and Social Studies). ELA classrooms were scored slightly higher on average than other

core courses in the frequency and quality of examples of three effective teaching practices

(Regard for Student Perspectives, Quality of Feedback, and Instruction Dialogue). Social Studies

classrooms received the lowest average ratings for observed frequency and quality of the other

two dimensions (Analysis and Problem Solving and Content Understanding). These observation

data mirror results from teachers’ self-report surveys about frequency of effective, standards-

based teaching practices.

Table 21. Mean CLASS Scores by Content Area Classroom

CLASS Construct

ELA

(n = 219)

Math

(n = 178)

Science

(n = 177)

Social

Studies

(n = 145)

Overall

Mean

(n = 702)

Regard for Student Perspectives 4.28 3.76 3.80 3.87 3.94

Content Understanding 4.11 4.12 4.16 3.53 3.98

Analysis and Problem Solving 2.95 2.90 2.92 2.37 2.80

Quality Feedback 3.98 3.87 3.72 3.42 3.75

Instructional Dialogue 3.70 3.25 3.35 3.23 3.38

Source: CLASS Observation Protocol

Finally, items on the RttT Omnibus Survey asked teachers to indicate how often they are

implementing instructional practices linked to the RttT priority of formative assessment. From

last year to this year, Omnibus results were similar for the most part, other than very slight

decreases in teachers’ estimates of their frequency of time spent on formative assessment

practices. These differences could be accounted for by the sizeable increase in sample size

between years (Table 22, following page). Teachers reported that they are regularly highlighting

learning targets (91%) and modifying instruction based on classroom assessments (84%), but

rarely giving students opportunities to give input on assessment assignments (25%) or to assess


March 2013


their peers (32%). Teachers may need additional supports to incorporate some of the more

student-directed aspects of formative assessment in their classrooms.

Table 22. Teacher Estimations of Time Spent on Formative Assessment Practices

How often do you do each of the following?

2011

Percent

Once/Twice a

Week or Almost

Daily

(n = 1,709)

2012

Percent

Once/Twice a

Week or Almost

Daily

(n = 10,011)

I use checklists when gathering information about student learning. 64% 60%

I use rubrics for assessing my students. 56% 54%

I write learning targets on the board and go over them with my

students. 85% 84%

I provide students specific information (without using grades or

rubrics) about where they are in meeting the learning targets. 78% 77%

I plan or modify classroom instruction based on the information I

receive from classroom assessment. 92% 91%

I give students opportunities to self-assess and set goals for future

learning. 62% 59%

I give students opportunities to reflect on and share their learning

progress with others. 65% 64%

I give students opportunities to provide input on assessment design. 35% 25%

I give students opportunities to formatively assess their peers. 26% 32%


These aggregated results provide a cursory assessment of Year 1 progress toward integrating

Common Core and Essential Standards and formative assessment practices into classroom

learning strategies. In future reports, these baseline results will be compared across years to

assess annual progress toward transition to the new standards and practices.


March 2013


Next Steps for the Professional Development Evaluation

This report reflects the status of the RttT professional development evaluation as of September

2012 and provides detailed information concerning near- and longer-term impacts of statewide

face-to-face RttT professional development efforts at the local level. The Evaluation Team

collected and analyzed relevant data from all 115 LEAs and identified a purposeful sample of 27

schools to examine more deeply the extent to which LEA and school staff built capacity to

provide high-quality professional development, support shifts in local organizational conditions

to support RttT priorities, and encourage changes to instructional practice.

The work by both the Professional Development Implementation Team and the Evaluation Team

is well underway, and the activities for both during the next two years of the RttT grant will be

challenging. In this section, we briefly summarize some of the major next steps for the

professional development evaluation, noting that some require close collaboration with the

NCDPI RttT Professional Development Implementation Team.

Addressing Study Limitations

The Evaluation Team is making every effort to address study limitations.

CLASS: For the purposes of this study, this data will be used to describe teachers’ transition

to the new state content standards (Common Core and Essential Standards), as evidenced by

effective instructional practices required for the teaching to the new standards. CLASS data

will not be used to evaluate teachers or track individual teaching practices over time.

Teacher Professional Development Survey: Once identified, the additional agree option was

corrected for the items about Quality of Professional Development and LEA Capacity. The

Evaluation Team will continue to use validity and reliability analysis results to streamline

and improve surveys.

Continued Evaluation of the RttT Annual Professional Development Cycle

The Evaluation Team will continue to document the status of the state’s face-to-face and online

RttT professional development efforts, addressing questions under Evaluation Question 1 (State

Strategies). This part of the study is intended to contribute to the RttT Professional Development

Implementation Team’s ongoing efforts to provide the most supportive and useful professional

development possible for teachers and administrators.

The Evaluation Team will modify and revise the evaluation plan as we move into the final years

of RttT to shift the focus to local outcomes. The next major report, in Fall 2013, will consider

short-term outcomes in more detail and report changes from baseline for intermediate outcomes,

addressing questions under Evaluation Questions 2 and 3 (Short-Term Outcomes and

Intermediate Outcomes). Reports in the final year of the grant will also address the impact on

teachers and, to the extent possible, on students, addressing questions under Evaluation Question

4 (Impacts on Student Performance).


March 2013


The overall evaluation of the impact of state-level RttT professional development efforts centers

on collection and analyses of: (1) an annual RttT Omnibus Survey study, as a way to gauge

changes in awareness, attitudes, knowledge/skills, and practices of educators across the state; and

(2) a mixed methods longitudinal descriptive study to provide an in-depth look at the

implementation and impact of RttT initiatives at the local level, with a focus on a purposeful

sample of 27 school sites across the state. Both the Omnibus Survey and the longitudinal study

will continue through the remaining two years of the RttT grant and will be discussed in each

annual professional development evaluation report.

Continued Evaluation of the Distinguished Leadership in Practice (DLP) Principal Institutes

The Distinguished Leadership in Practice professional development initiative for principals is led

by the North Carolina Principals and Assistant Principals Association (NCPAPA). The

Evaluation Team has worked with NCPAPA to examine the implementation of the DLP program

and its impact on the effective leadership capacity of North Carolina principals. The first

formative evaluation report was successfully approved by North Carolina’s RttT Leadership

Team, the North Carolina State Board of Education, and the Governor’s Education

Transformation Commission in September 2012. This first report provided detailed information

about the delivery and quality of the DLP program, its alignment to key goals (e.g., improvement

of educator attitudes, knowledge, and skills), and data-informed recommendations for improving

the program. Data sources included participant surveys, interviews, and focus groups with a

purposeful sample of participants and key personnel; DLP documents such as participant

applications and artifacts; principal evaluation data; and administrative data. Annual evaluation

reports for DLP will be submitted in the summer to align with the DLP program calendar.

Continued Collaboration with NCDPI Staff

This evaluation requires close collaboration with the NCDPI team to ensure that the data

gathered provide the information needed to inform program improvement and formulate an

evaluation that meets the grant requirements, informs future policy and program decisions, and

contributes to the field of online professional development. With these goals in mind, we will

continue to adapt the data collection process to provide the most useful formative results to

NCDPI staff. The Evaluation Team has created and provided access to real-time reports from

online participant surveys available to NCDPI staff (sample report: go.ncsu.edu/2012sipostall).

The Evaluation Team meets with RttT professional development implementation teams to

discuss the evaluation plan, data collection, instruments, and results.


March 2013


References

North Carolina Department of Public Instruction. (2012).2011-12 North Carolina Public Schools

Statistical Profile. Available from http://apps.schools.nc.gov/statisticalprofile.

Teachstone (2012). The CLASS Tool Website. University of Virginia: Charlottesville, VA.

Available from http://www.teachstone.org/about-the-class/.

http://apps.schools.nc.gov/statisticalprofile

http://www.teachstone.org/about-the-class/


March 2013


Appendix A. RttT Professional Development Teacher Survey Items

Quality of PD

To what degree do you agree with the following statements about the professional development

experiences provided in your district this year?

Response Options: Strongly Disagree, Disagree, Slightly Disagree, Neither Agree nor Disagree,

Slightly Agree, Agree, Strongly Agree, Not Applicable

1. I have applied what I learned in professional development to my classroom

2. Helped me improve the way I teach

3. Has been differentiated to meet teacher participants’ needs

4. Been closely connected to my school’s improvement plan

District Capacity

In my district, leaders…



5. Have knowledge about high quality professional development defined by research and

national and state standards

6. Have the skills to plan and design high quality professional development

7. Have the skills to implement high quality professional development

8. Have the skills to select high quality professional development

9. Have positive attitudes about high quality professional development

10. Have a district-wide commitment to high quality professional development

11. Support Communities of Practice around high quality professional development

12. Provide opportunities for networking and support in high quality professional development

13. Have a district-level strategic plan for professional development in place

14. Align school-level professional development plans to district plans

15. Distribute responsibilities for leadership for professional development among multiple school

and district administrators

16. Use data from teacher performance evaluations to create individual professional development

plans for teachers

17. Aggregate data from teacher performance evaluations and use data to identify school/district

professional development needs

18. Use survey data to select, plan, and design professional development


March 2013


19. Use summative student data to select, plan, and design professional development

20. Use formative student data to select, plan, and design professional development

21. Provide access to high-quality online professional development opportunities

22. Extend and enhance on-site professional development through the use online communication

and resources

23. Support professional learning communities by providing access to web 2.0 tools such as

blogs, wikis, and social networking tools

24. Support professional learning communities by providing an online space to share ideas and

resources

25. Model effective use of web-based communication and collaboration tools to support

professional development

26. Provide support for users uncomfortable with online professional development opportunities

PD Coverage

To what extent do you feel that teacher knowledge and skills have been enhanced in each of the

following areas as a result of your participation in professional development?



27. Revised state standards

28. Instructional materials

29. Approaches to formative assessment

30. Use of technology

31. Strategies for teaching diverse student populations

32. Deepening content knowledge

33. Leadership development

34. Revised state assessments

35. Revised NC Teacher Evaluation Process

Common Core and Essential Standards

To what extent do you agree or disagree with each of the following statements?



36. I am aware of professional development opportunities on the revised (Common Core and

Essential Standards) State Standards.


March 2013


37. I know what students are expected to know, understand, and do in regards to the revised

(Common Core and Essential Standards) State Standards.

38. Administrators in my school know what students are expected to know, understand, and do in

regards to the revised (Common Core and Essential Standards) State Standards.

39. I know how teaching the revised (Common Core and Essential Standards) State Standards

will differ from the current North Carolina Standard Course of Study.

40. Administrators in my school know how teaching the revised (Common Core and Essential

Standards) State Standards will differ from the current North Carolina Standard Course of

Study.

Math

How much of the mathematics instructional time do students use to engage in the following

tasks?

Response Options: Never, A Few Times a Year, Once or Twice a Month, Once or Twice a

Week, Almost Daily

1. Make sense of problems

2. Persevere in solving problems

3. Reason abstractly

4. Reason quantitatively

5. Construct viable arguments

6. Critique the reasoning of others

7. Model with mathematics

8. Use appropriate tools strategically

9. Attend to precision

English Language Arts

How much of the ELA instructional time do students use to engage in the following tasks?


Week, Almost Daily

1. Read increasingly complex texts with increasing independence

2. Analyze and synthesize sources

3. Present careful analysis, well-defended claims, and clear information

4. Gain listening skills

5. Respond to and challenge their peers with relevant follow-up questions and evidence

6. Acquire knowledge of general academic vocabulary


March 2013


7. Draw evidence from texts to support their response

8. Read deeply to gain knowledge from texts

Science

How much of the science instructional time do students use to engage in the following tasks?


Week, Almost Daily

1. Develop problem-solving skills through investigations

2. Work in small groups

3. Make predictions that can be tested

4. Make careful observations

5. Use tools to gather data (e.g. calculators, computers, graduated cylinders, scales and meter

sticks)

6. Measure with accuracy

7. Conduct multiple trials of an investigation to test a prediction

8. Keep accurate records of investigation trials

9. Recognize patterns in data

10. Create reasonable explanations of results of an experiment or investigation

11. Choose the most appropriate mechanism to express results (e.g. scientific language,

drawings, models, charts or graphs)

12. Utilize appropriate safety procedures when conducting scientific investigations

13. Participate in hands-on-activities

14. Complete activities with a real-world context

15. Engage in technological design investigations

16. Conduct scientific investigations

Social Studies

How much of the Social Studies/History instructional time do students use to engage in the

following tasks?


Week, Almost Daily

1. Demonstrate chronological thinking

2. Ask questions that historians ask

3. Explore changes in communities and regions over time


March 2013


4. Recognize and appreciate the contributions of diverse cultural groups

5. Explain why people can describe the same event differently

6. Draw connections between contemporary issues and their historical origins

7. Transfer understanding from the state to the national level

8. Articulate the implications of increased global interactions

9. Recognize and interpret the “lessons of history”

10. Identify pivotal moments in world history that shaped the development of contemporary

societies

11. Use geography to understand current global conditions

12. Identify patterns of continuity and change

13. Articulate the roles of the government

14. Learn to make responsible financial choices in spending and saving

15. Use maps, charts, and graphs, and other geographic tools

16. Compare multiple perspectives and interpretations of the same issue, time period, etc.

17. Analyze primary documents and other artifacts

18. Actively participate as a citizen

19. Construct historical narratives

20. Analyze cause and effect relationships


March 2013


Appendix B. LEA Professional Development Coordinator Interview Protocol

Overall Professional Development - When we say “professional development,” in addition to the

traditional face-to-face workshop, we also want you to think of less-traditional supports like

blended and online webinars, technical assistance, collaboration, coaching, and PLCs.

1. In what way does collaboration play a role in preparing your school/districts for the roll out

of Common Core/ Essential Standards?

Who in your district helps you to coordinate professional development training around

the Common Core/ Essential Standards?

2. How have the professional development strategies at your school/district been affected by the

new Common Core/ Essential Standards?

How has professional development changed as a result of DPI’s statewide plan for

professional development? (Picture)

Do you feel that the suggested professional development is appropriate for your

school/district?

Describe how the standards are being integrated into professional development strategies

at the district level.

What policies or procedures are in place to support ongoing professional development

around the Common Core/ Essential Standards?

3. How is technology being integrated into the professional development training efforts in your

district?

What webinars or NC Education online learning modules have you participated in?

o Describe your experience using these modules or webinars.

o In what ways do you anticipate the learning modules will be useful for teachers?

Challenging?

o Do you think they will impact teaching practices?

How do you plan on integrating these DPI online professional development resources

into your district professional development plan?

o How will your district monitor progress and completions?

o How will you assess their effectiveness/ learner mastery?

o Will you offer specific incentives for completion of the modules such as CEUs or

certificates of completion?

Have you provided your teachers any additional online resources to support RttT related

professional development?

o Would you say that their experience with online training material has been useful?


March 2013


4. The NC RttT Grant has several priority areas. What effort or support has been provided to

your district to inform staff about:

Successful Transition to New Standards (Essential and Common Core)

Formative and Summative Assessment

Use Data to Improve Instruction

Effective Utilization of the NCTEP

Effective Use of Technology for Teaching and Learning


March 2013


Appendix C: Principal Interviews Interview Protocol




Since our last interview:

1. What if any changes have you noticed in district strategies and policies regarding

professional development?

2. Have there been any changes to your approach in determining what professional

development is appropriate for your district/school?

a. Do you use results from the NC Teacher Evaluation Process?

3. Have you noticed any changes in the resources available to you to implement professional

development?

4. Have there been any changes in the way you access or locate appropriate professional

development for your district/school?

a. Do you access professional development through DPI? Other state

agencies/organizations? Vendors?

5. Have you found more efficient ways to implement professional development?

6. How do you determine who should attend/participate in professional development

opportunities?

7. What support do you provide to your staff so they can attend professional development?

8. Have your expectations for your staff (teachers and school/district leaders) to participate in

professional development changed?

9. Are you aware of the DPI webinars and the online learning modules offered through NC

Education?

a. If so, which ones have you accessed?

i. How would you describe your experience using these modules and/or webinars?

ii. Do you think they will be useful for teachers?

iii. Do you think they will impact their practice?

b. Do you plan on integrating these DPI online professional development resources into

your district or school professional development plan? If so, how?

i. How will your district or school monitor progress and completions?

ii. How will your assess their effectiveness/ learner mastery?


March 2013


iii. Will you grant CEUs for certificated of completion?

c. Do your teachers have access to any additional online resources to support RttT related

professional development? If so, please describe.

i. Would you say that their experience with online training material has been useful?

10. The NC RttT Grant has several priority areas. Since we last spoke, what effort or support has

been provided to your district to inform staff about:

a. Successful Transition to New Standards (Essential Standards and Common Core)

b. Formative and Summative Assessment (NCFALCON)

c. Use Data to Improve Instruction

d. Effective Utilization of the NC Teacher Evaluation Process (NC TEP)

e. Effective Use of Technology for Teaching and Learning


March 2013


Appendix D. Teacher Focus Group Protocol




1. What professional development resources have been provided from your school/district in

preparation for the roll out of common core/essential standards?

What do you anticipate will be the biggest challenges in transitioning to the new

standards?

What do you anticipate will be the greatest benefit in transitioning to the new standards?

2. In what way does collaboration with other educators play a role in professional development

training around common core/ essential standards at your school/district?

3. How are updates/trainings around common core/essential standards communicated between

the teachers, school administrators and district staff?

4. How is technology being integrated into the professional development training efforts for the

new state standards?

Are you aware of the online professional development opportunities being offered by

DPI? (i.e. NC Education Modules)

If yes…

What, if any NC Education modules or webinars have you used?

o How would you describe your experience with these modules?

o Did you find them useful?

o In what ways have they impacted your practice?

What, if any additional online professional development opportunities are being offered

by your school or district?

o Describe your experiences with these local online professional development

opportunities.

o In what ways has your experience with local online professional development

opportunities been useful? Challenging?

5. The NC RttT Grant has several priority areas. What effort or support has been provided to

your school to inform staff about:

Successful Transition to New Standards (Essential and Common Core)

Formative and Summative Assessment

Use of Data to Improve Instruction


March 2013


Effective Utilization of the revised North Carolina Teacher Evaluation Process

(NCTEP)

Effective Use of Technology for Teaching and Learning

Contact Information:

Please direct all inquiries to Jeni Corn, Ph.D.

[email protected]

919-513-8527

© 2013 Consortium for Educational Research and Evaluation–North Carolina

mailto:[email protected]