Teacher Evaluation in Practice: Implementing Chicago's REACH Students

7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students

1/44

RESEARCH REPORT SEPTEMBER 2013

Teacher Evaluation in Practice

Implementing Chicagos REACH Students

Susan E. Sporte, W. David Stevens, Kaleen Healey, Jennie Jiang, and Holly Hart


2/44

1 Executive Summary

3 Introduction

Chapter 1

9 The Classroom Observation Process

Chapter 2

15 The Use o Student Growth inEvaluation

Chapter 3

21 Training and Communication

Acknowledgements

The authors grateully acknowledge the support o the Joyce Foundation and its continuing interest in this

important line o work. This analysis based on one districts experience with a dramatically dierent teacher evalu-

ation system could not have happened without Joyces steadast support and long term commitment. In addition

we would like to acknowledge the support o Chicago Public Schools and the Chicago Teachers Union. We have

gained particular insights rom Matt Lyons, Paulette Poncelet, Sheila Cashman, Elizabeth Press, Didi Schwartz,

Amanda Smith, Susan Kajiwara-Ansai, and Meghan Zeran rom CPS central oce and rom Carol Care o the

CTU as well as other teachers rom the CPS-CTU Joint Committee who have provided valuable eedback as the

work has progressed.

In addition we thank the teachers and administrators o Chicago Public Schools who shared their thoughts

through multiple surveys, and those who provided their valuable time in one-on-one interviews. This report

was made possible through their participation. We also learned rom those individuals serving as Instructional

Eectiveness Specialists and thank them or the extra time they spent sharing with us.

We are also indebted to the Research Assistants who helped in countless ways to make this report possible:

Patrick Wu, Elc Estrera, Jen Cowhy, Catherine Alvarez-McCurdy, Josie Glore, and Gabrielle Friedman. Their con-

tributions have been invaluable. We thank members o the University o Chicago Consortium on School Researchs

Steering Committee who commented on an earlier drat o this reportespecially Peter Godard, Karen Lewis, and

Luis Soria. It is always humbling to have our work critiqued by those with on-the-ground experience and we thank

them or sharing that experience with us. Their thoughts helped shape this nal product. Finally, we owe a deep

debt o gratitude to members o the UChicago CCSR sta who worked with us as thought partners and readers

through numerous iterations to arrive at this nal report: The communications sta, Emily Krone and Bronwyn

McDaniel, who kept us ocused and oered excellent eedback; ellow researchers Elaine Allensworth, Jenny

Nagaoka, Melissa Roderick, Penny Sebring, Marisa de la Torre, and Stuart Luppescu; and our superb nal technical

readers Lauren Sartain and Paul Moore. Our deep discussions with all o them have added to this report as well as

our understanding o the promise and challenge o changing the teacher evaluation system in Chicago.

Chapter 4

25 How Principals Manage Classroom

Observation Workloads

Chapter 5

29 Questions to Consider

33 Reerences

35 Appendices

37 Endnotes

TABLE OF CONTENTS

This report was produced by UChicago CCSRs publications

and communications sta: Emily Krone, Director or Outreach

and Communication; Bronwyn McDaniel, Communications and

Research Manager; and Jessica Puller, Communications Specialist.

Graphic Design: Je Hall Design

Photography: Cynthia Howe and David Schalliol

Editing: Ann Lindner

09.2013/pd/[email protected]


3/44

Executive Summary

Executive Summary

This report ocuses on the perceptions and experiences o teachers and

administrators during the rst year o REACH implementation, which was

in many ways a particularly demanding year. These experiences can be

helpul to CPS and to other districts across the country as they work to

restructure and transorm teacher evaluation.

Historically, teacher evaluation in Chicago has allen

short on two crucial ronts: It has not provided adminis-

trators with measures t hat dierentiated among strong

and weak teachersin act, 93 percent o teachers were

rated as Excellent or Superiorand it has not provided

teachers with useul eedback they could use to improve

their instruction.1

Chicago is not uniqueteacher evaluation systems

across the country have experienced the exact same

problems.2Recent national policy has emphasized

overhauling these systems to include multiple measures

o teacher perormance, such as student outcomes, and

structuring the evaluations so they are useul rom both

talent management and teacher proessional develop-

ment perspectives. Principals and teachers need an

evaluation system that provides teachers with speciic,

practice-oriented eedback they can use to improve their

instruction and school leaders need to be able to identiy

strong and weak teachers. Required to act by a new state

law and building o lessons learned rom an earlier pilot

o an evidence-based observation tool,3 Chicago Public

Schools (CPS) rolled out its new teacher evaluation

systemRecognizing Educators Advancing Chicagos

Students (RE ACH Students)in the 2012-13 school yea r.

The REACH system seeks to provide a measure o

individual teacher eectiveness that ca n simultane-

ously support instructional improvement. It incorpo-

rates teacher perormance ratings based on multiple

classroom observations together with student growth

measured on two dierent types o assessments.

Wh ile the pract ice o usi ng cl assroom observations

as an evaluation tool is not completely new, REACH

requires teachers and administrators to conceptualize

classroom observations more broadly as being part o

instructional improvement eorts as well as eva luation;

evaluating teachers based on student test score growth

has never happened beore in the district.

REACH implementation was a massive undertak-

ing. It required a large-scale investment o time and

energy rom teachers, administrators, CPS central oice

sta, and the teachers union. District context played an

important role and provided additional challenges as the

district was introducing other major initiatives at the

same time as REACH. Furthermore, the school year

began with the irst teacher strike in CPS in over 25

years. Teacher evaluation was one o several contentious

points in the protracted negotiation, and the speciic

issue o using student growth on assessments to evaluate

teachers received considerable coverage in the media.

This study uses data collected rom a ll 2012

through spring 2013, including:

Two surveys o all 1,195 principals and assistant

principals, administered in December 2012 and

Apri l/May 2013, re spec tively


4/44

UCHICAGO CCSR Research Report | Teacher Evaluation in Practice

2

Two surveys o teachers, one administered in

January 2013 to a sample o 2,000 classroom

teachers and one administered in March 2 013

to all teachers in the district

Interviews with a random sample o 31 classroom

teachers and six principals rom six schools,

conducted in spring 2013

Interviews with nine central oice sta members

(Instructional Eectiveness Specialists), conducted

in November 2012

Summary o Main Findings:

Teachers and administrators nd the observation

process useul or improving instruction

Overwhelming majorities o teachers and admin-

istrators believe the observation process supports

teacher growth, identiies areas o strength and

weakness, and has improved t he qu ality o

proessional conversations between them.

Most administrators eel conident in their ability

to gather evidence and assign ratings; a large

majority o teachers believe their evaluator is air

and unbiased and able to assess their instr uction.

Some teachers expressed concern that classroom

observation ratings are too subjective to be usedin high-stakes evaluations, while others eel appre-

hensive about revealing instr uctional weaknesses

or ear o being penalized on their evaluations.

Teachers are hesitant about the use o student

growth on assessments to evaluate their

classroom perormance

Over hal o teachers surveyed believe REACH relies

too heavily on student growth.

Special education teachers are particularly criticaland nd the assessments to be inappropriate

measures o their students learning and their

instruction.

Communication with teachers is an area or

improvement; administrators want support

on coaching and providing useul eedback

The requency and quality o training and

communication received by teachers varies widely.

Teachers are conused about how student growth

actors into their inal rating. Both teachers and

administrators need clarity about score ca lculations

and how they will be used or personnel decisions.

Most administrators list coaching and providing

useul eedback as high priorities or their own

proessional development.

REACH places demands on administrator

time and capacity

Administrators reported spending about six hours

per ormal observation cycle, including the observa-

tion, pre- and post-observation conerences, and

data management. Based on the amount o time

administrators reported spending on observations,

and the average number o observations perormed,

the ty pical elementary school administrator spent

approximately 120 hoursor two ull weekssolely

on observations that were part o the teacher evalu-

ation system. The typical high school administrator

spent approximately three ul l weeks.

Administrators are expected to train teachers about

the system, conduct classroom observations, hold

meaningul conversations with teachers about t heir

instruction, and complete required paperwork while

balancing their other job responsibilities.

This report is the irst in a series o studies

on Chicagos REACH teacher evaluation system.

Subsequent work will investigate the consistency in

observation ratings, the multiple measures o student

growth, and the relationships among these variables.

As the init iati ve continue s to unold, uture work w ill

also examine changes in these measures over time.


5/44

Introduction

Introduction

In the all o 2012, Chicago Public Schools (CPS) instituted a sweeping

reorm o its teacher evaluation system with the introduction o REACH

Students. REACH Students replaces CPSs ormer 1970sera checklist

policy by incorporating a detailed classroom observation process and

student growth measures into teachers eectiveness scores (i.e., ormal

or summative evaluation ratings).4

With thi s pol icy, Ch icago joins other states a nd

districts across the country in developing new systems

to evaluate teacher perormance. More than 40 states

now incorporate student test scores or other achieve-

ment measures into their teacher evaluations.5 Over

the next ew years, several la rge urban districts (e.g.,

Los Angeles, Philadelphia, and New York) will be pilot-

ing or implementing similar new teacher evaluation

systems required by their states.

This report provides an initial look at the irst-year

implementation o REACH (Recognizing Educators

Adva ncing Chic ago) St udents (herea ter reerre d to as

REACH). Recent reports on teacher evaluation have

highlighted the problems that systems like Chicagos

attempt to correct, but there is still much to learn about

districts implementation experiences and their early

successes and challenges. We begin by describing the

REACH evaluation system and the speciic questions

that guided our study.

Purpose and Design o REACHRecent eorts to revamp teacher evaluation systems

relect the education ields increasing shit in ocus

rom schools to individual teachers.6 A growing number

o studies are examini ng how student learning is re-

lated to teacher eectiveness. This work shows student

achievement gains va ry signiicantly across teachers.

Furthermore, teacher eectiveness accounts or more

var iation in stud ent out comes tha n any other school

actor.7 Policymakers have responded to these research

indings: ederal policy under the U.S. Department o

Educations Race to the Top grant competition encour-

ages states to identiy strong and weak teachers by in-

corporating multiple measures o teacher perormance

in state evaluation requirements.8 Combined, develop-

ments in education research and policy have put teacher

eectiveness ront-and-center o eorts to improve

students educational outcomes.

The possibility o receiving a ederal Race to the Top

grant prompted the Illinois State Board o Education

to pursue key goals or providing students with access

to high-quality teacher and leaders, and it incentivized

Illinois legislators to pass the Perormance Evaluation

Reorm Act (PERA) in 2010. PERA requires every dis-

trict in Illinois to adopt new teacher evaluation systems

that assess both teacher practice and st udent growth. 9

The teacher practice measures required by PERA must

include multiple ormal classroom observations, as

well as suppor t or teac her i mprovement. For student

growth, the law deines various quali ying assessment

types a nd combinations o assessments that must be

used. Teacher perormance and st udent growth ratings

must then be combined to create a single, summative

rating o teacher perormance.

To comply with PERA requirements and to build

o a generally successul pilot o an ev idence-based


6/44


4

observation rubric (see CPSS Experiment with Teacher

Evaluation: EITP, p. 8), CPS rolled out its new teacher

evaluation systemREACHin the 2012-13 school year.

The main components o REACH in 2012-13 include:

Multiple classroom observations: Non-tenured

teachers must be observed our times per year, and

observations must last or at least 45 minutes and

include a pre- and post-observation conerence.

REACH requires administrators to provide eedback

to teachers ater each observation.

An explicit observation rubric: REACH utilizes

a modiied version o the Charlotte Danielson

Framework or Teaching.10 In this rubric teachers are

rated on our areas, or domains, o teaching practice:

Planning and Preparation, Classroom Environment,

Instruction, and Proessional Responsibilities. Each

o the domains is urther broken down into 4-5

components in which expectations or each level

o perormance are described in detail.

Trained evaluators: REACH requires all administra-

tors to be certiied by completing a series o training

modules and passing two a ssessments. It urther

employs trained specialists t hat work with adminis-

trators on calibration and assigning evidence-based

ratings aligned with the r ubric.

Student growth measures: REACH utilizes two di-

erent measures o student growth (Perormance

Tasks and either value-added or expected gains).

Althou gh R EACH is i ntended to provide a more accu-

rate measurement o teacher practice, CPS has been clear

that the system should also be a vehicle or proessional

growth. The CPS observation rubric (herea ter reerred

to as the Fra mework) provides a common language

about what constitutes eective teaching and a struc-

ture or having conversations ocused on supporting

instructional improvement (see Appendix B). Recent

research on such process-based observations systems

suggests that they can lead to improved student learn-

ing.11 Furthermore, while test score data are intended to

provide an additional measure o t eacher eectiveness,

they are a lso intended to inorm teachers choices about

appropriate instructional content or their students.

REACH implementation was a massive undertaking.

It required a large-scale investment o time and energy

rom teachers and administrators alikein the orm o

training or administrators to be certiied as observ-

ers, more requent and time-intensive observations and

conerences or both teachers and administrators, andoverall training on a new and complex system. By the

end o this year, the observation process had resulted in

over 36,000 observations or about 6,00 0 non-tenured

teachers and 13,000 tenured teachers. REACH also

required the district to create a whole new set o assess-

ments since many teachers do not teach in grade levels

or subject areas that are captured on typical standard-

ized assessments. In order to link students and teachers

to provide accurate student growth inormation, the

CPS central oice had to redesign the way data onteachers and students are collected.

TABLE 1

CPS School and Personnel Statistics (2012-13)

Schools* 578

Elementary Schools 472

High Schools 106

Non-Tenured Teachers 5,743

Tenured Teachers 15,109

Administrators** 1,195

Source: CPS Stats and Facts, Administrative records

* Does not include charter or contract schools

** Only includes principals and assistant principals

The 2012-13 school year was particularly diicult

time to launch such a large-scale and complex teacher

evaluation system: The school year began with the irst

teacher strike in more than two decades; the CEO o

CPS resigned in October, ushering in the third leader-

ship change in our years; all schools had a longer day

and year; and CPS began transitioning to the Common

Core State Standards or teaching and learning. On top

o all o this, debates about school closings, enrollment

declines, and budget shortalls began in the all. A series

o heavily attended and emotional public hearings were

held throughout the year, and a controversial decision

was made in t he spring to c lose 49 school s.


7/44

Introduction

Guiding QuestionsThe increased attention to teacher evaluation rom

policymakers and practitioners has been accompanied

by increased attention rom researchers seeking to

evaluate implementation o these new systems. Many

studies have ocused on technical aspects, such asthe reliability o t he measurement tools.12 Another

important, but smaller, body o work has exa mined

the use o new teacher evaluation systems in schools

and districts.13 Building on this early research, this

report provides inormation on the irst year o REACH

implementation, answering questions about teachers

and administrators perceptions o the system a nd

their experiences with the new system. The speciic

questions and issues explored in this report include:

QUESTION 1: What are the benets and

drawbacks o observation systems designed

or both teacher development and evaluation?

One o the beneits o using classroom observations

in evaluation systems is that they have the potential

to meet schools dual needs o supporting proes-

sional growth and dierentiating teacher practice.14

Observations can create structures or providing

teachers with timely and individualized eedback on

their classroom practice. This inormation can guide

coaching and proessional development activities, as

well as help t eachers de velop goals or improvement. I n

addition, observation ratings provide adm inistrators

with st and ard ized and deen sible evidence or ma king

personnel decisions.

Yet, using observations or both purposes may also

create a number o tensions. One study suggests that

some teachers may be less likely to seek instructional

support rom administrators i exposing their weak-

nesses could result in a poor evaluation.15 Furthermore,

teachers may not respond positively to encouragement

rom administrators ater receiving low ratings or

disciplinary actions rom them.16 Finally, classroom

evaluators who are responsible or supporting teacher

growth and ormally assessing eectiveness may intro-

duce bias into the accountability process.17 In short, i

not implemented well, the beneits o using classroom

observations may devolve into dueling purposes with

each cancelling the beneits o the other.

UChicago CCSRs study o CPSs earlier pilot pro-

gram, which was ca lled the Excellence in Teaching

Pilot, ound that teachers and principals thought their

discussions about instruction were more relective and

objective using the Danielson Framework than the CPS

checklist.18

Observations conducted under the pilot,however, did not count toward teachers oicial evalu-

ation score. In Chapter 1, we ask: How air and useul

do teachers and administrators ind R EACH classroom

observations as a means o improving instruction?

Does using school administrators as both coaches

and evaluators raise any concerns or challenges?

QUESTION 2: How do teachers view the use o

student growth on standardized assessments

in their evaluation?

The incorporation o student growth measures into

teachers evaluations has been a contentious issue,

both in Chicago a nd nationally. While supporters

maintain teachers should be held accountable or stu-

dent learning, critics contend that metrics designed to

assess student progress are poor measures o teacher

perormance.19 Additionally, opponents ear that

adding stakes to student assessments increases the

likelihood that teachers will nar row their curriculum or

teach to the test so as to avoid a negative evaluation.

Despite these issues, states and districts have moved

orward with including student growth measures in

teachers evaluations.

Addre ssin g teacher and admini str ator skept icism o

student growth measures is critical or leveraging the

ull potential o the system to improve instruction. In

Chapter 2, we ask: To what extent do teachers perceive

student growth measures can provide an accurate as-

sessment o their perormance? How, i at all, are teach-

ers using the assessment data produced by REACH?

QUESTION 3: What are the successes and chal-

lenges related to training and communication?

Wh ile RE ACH add resses many o the lim itations o t he

previous teacher checklist system, thoughtul design is

not enough to guara ntee success.20 Beore REACH can

improve teacher evaluation or instructional practice,

it irst has to move rom a written policy document to

a system embedded in the work o teachers and school


8/44


6

administrators. Implementation is critical to achieving

intended outcomes.21

The task o implementing a teacher evaluation sys-

tem o this scale and complexity should not be under-

estimated. REACH involves over 20,000 teachers and

other school sta and 1,200 admin istrators in nearly600 schools. Principals and assistant principals had to

be certiied and trained on using the new observation

rubric. Teachers had to be inormed about the goals o

the new system, trained on how to engage in the new

observation process, and taught how their summative

evaluation score would be calculated. Observations and

the pre- and post-observation conerences had to be

scheduled and completed.

Understanding the experiences o teachers and

administrators as they implemented such a complexand time-intensive systemin addition to all their

other responsibilitiesis a critical irst step toward

understanding any potential eects that REACH

might have. I teachers and administrators are not

inormed o REACHs goals and do not understand its

var ious element s, they m ay not implement the poli cy

as intended. Insuicient training and resources are

reasons or implementation ailure. 22 In Chapter 3,

we a sk: How knowledgeable w ere t eachers a nd ad min-

istrators about REACH? How did they describe their

training ex periences? What aspects o implementation

did participants identiy as needing improvement?

QUESTION 4: How do principals understand and

describe their capacity to manage classroom

observation workloads?

In the la st decade, principals have been increasingly

called upon to be instr uctional leaders in their schools,

especially through supporting eective instructional

practices. 23 Given this emphasis on principals as in-

structional leaders, many assume that it is the principal

who s hould be r esponsible or conduc ting obs erv ation s

and evaluating teacher practice.

It is not clear, however, whether principals have the

time and capacity to manage the observation workload

created by new evaluation systems. To increase the

reliability o ratings, most systems call or teachers to

be observed multiple times a year. Each observation

typically involves scheduling and conducting the obser-vation, w riti ng up evidence a nd enteri ng it into a dat a-

base, having pre- and post-observation discussions with

teachers, and coaching teachers on areas or improve-

ment. The entire process or a single teacher can take

several hours. While assistant principals in CPS also

became certiied evaluators, it still ell to the principals

to ensure that all o the observations and pre- and post-

conerences required by REACH were conducted.

Previous studies conducted by UChicago CCSR

researchers have highlighted some o the capacity is-sues created by the introduction o new teacher

evaluation systems. For example, workload demands

contributed to lower engagement in the new system

or some principals, while others reported giving less

attention to tenured teachers in order to complete all

o their required eva luations.24 In Chapter 4, we ask:

How much time did administrators spend on classroom

observations during the irst year o REACH? How do

they eel about the demands the new REACH system

places on them?

In This ReportChapters 1 and 2 o this report describe the observation

and student growth elements o REACH and pro-

vide par ticipa nts perce ptions ab out t he va lue o thi s

initiative as both an evaluation and development tool.

Chapters 3 and 4 describe participants experiences

with implement ation, oc using on commu nication,

training, and time demands. Final ly, in Chapter 5, we

present some questions to consider as implementation

continues. Additional reports in th is series will inves-

tigate observation ratings, student growth ratings, and

the relationship between them.


9/44

Introduction

What Goes Into a Teachers Evaluation Score?

A teachers REACH summative evaluation score is

comprised o a teacher practice score and up to two

measures o student growth. The teacher practice

component consists o classroom observations com-

pleted by a certied administrator utilizing the CPS

Framework or Teaching, a modied version o the

Danielson Framework or Teaching. Student growth

measures are discussed in detail in Chapter 2. In

2012-13 only non-tenured teachers were to receive a

summative evaluation score.

Teacher Practice: CPS Framework for Teaching

Student Growth: REACH Performance Tasks

Student Growth: Value-Added

75%

10%

15%

Elementary Teachers in Tested Subjects/Grades

(Receive individual value-added)

75%

15%

10%

Elementary Teachers in Untested Subjects/Grades

(Receive schoolwide value-added in literacy)

Source: Chicago Public Schools

90%

10%

High School Teachers in Core Subject Areas

100%

High School Teachers in Non-Core Subject Areas


10/44


8

Research Activities

To answer our questions we used multiple sources o inormation, including surveys and interviews. Surveys pro-

vide a broad picture o participants perceptions; interviews provide deeper insights into participants experiences.

Winter 2012-13 Surveys: We surveyed all 1,195 prin-

cipals and assistant principals in December 2012,

receiving 733 responses (a 61 percent response

rate). We surveyed a random sample o 1,000

non-tenured teachers and 1,000 tenured teachers

in January 2013.A We received 901 responses (a 45

percent response rate). The entire content o this

survey administration was related to REACH.

Spring 2013 Surveys: We included survey items as

part o CPSs annual My Voice, My School survey.

This survey was administered to all teachers in

March 2013 and had a response rate o 81 percent.

Then, we surveyed all principals and assistant

principals in April/May 2013, receiving 687 respons-es (a 57 percent response rate). Some questions

were the same as in the winter 2012 survey to

gauge changes in perception; others were dierent

because the initiative was more mature. Survey

content was shared with other topics.

Spring 2013 Principal and Teacher Interviews:

We randomly selected three high schools and

ve elementary schools or our interview sample.

We then randomly selected teachers rom within

those schools to interview. We were able to

interview six principals and 31 classroom teachers

rom six schools.

Fall 2012 CPS Central Oce Staf

(Instructional Efectiveness Specialists)

Interviews: We interviewed nine specialists

(about hal o the sta in this position) in

November 2012. These specialists werecharged with providing technical assistance

to administrators in conducting classroom

observations.

CPSs Experiment with Teacher Evaluation: EITP

Between 2008 and 2010 CPS implemented the

Excellence in Teaching Pilot (EITP), a pilot teacher

evaluation program that used the Charlotte Danielson

Framework or Teaching to guide the evaluation o

classroom instruction. EITP provided an alternative

system to the teacher evaluation checklist CPS had

used or 30 years. Over the two-year period, a total

o 100 elementary schools participated in the pilot.

CPSs current REACH evaluation system

resembles EITP in many ways. Like the pilot, trained

administrator evaluators observe teachers classroom

instruction using a modied version o the Danielson

Framework. Some observations are unannounced

and others are planned in advance and include a pre-

and post-observation conerence. The new system,

however, diers rom EITP on several importantdimensions. First, although the pilot had no stakes

attached, REACH is the ocial evaluation system or

non-tenured teachers in its rst year, and will expand

to include stakes or all teachers over time. Second,

while administrator training or the smaller-scale pilot

was done in-person, training or the new system was

provided statewide via an online platorm. And nally,

the pilot provided measures o perormance based

only on observations, while the new system includes a

student growth component or all teachers regardless

o the grade or subject they teach.

UChicago CCSRs two-year study o the pilot ound

most principals and teachers were supportive o EITP

and ound it benecial or their practice. Specically,

principals and teachers reported using the Danielson

Framework and evidence rom classroom observations

made their conversations about instruction more

objective and refective. In addition, the study ound

principals ratings o teachers were both valid and

reliable.B The pilot also uncovered some challenges.

For example, many principals lacked the instructional

coaching skills required to have deep discussions about

teaching practice. Where principals were less procient

at conerencing with teachers, teachers were less posi-tive about the new system and more doubtul o their

principals ability to use the Framework accurately or

rate them airly. A later ollow-up study o EITP ound

that the pilot had a positive eect on both reading

and math scores. Higher-achieving schools and those

serving ewer low-income students were the primary

beneciaries.C


11/44

Chapter 1 | The Classroom Observation Process

The Classroom ObservationProcess

CHAPTER 1

The main element o the REACH evaluation system is

the observation process used to rate teacher practice.

The process is centered around the CPS Framework or

Teaching (the Framework), a classroom observation

rubric based on the Cha rlotte Danielson Framework

(see Appendix B). RE ACH also est abli shes a set o pro-

cedures or how evaluators should conduct classroom

observations, collect evidence about what was observed,

and discuss the evidence and ratings with teachers

(see Table 2).The teacher practice component is intended to serve

two unctions. Drawing on its roots in the Danielson

Framework, the classroom observation process is

structured to provide teachers with in ormation they

can use to improve their teaching practices. It includes

a pre- and post-observation conerence to create a

orum or evaluators to provide constructive eedback

to teachers on their practice and oer support or

improvement. In addition, the teacher practice compo-

nent is intended to provide school administrators with

a means to evaluate the instructional eectiveness o

TABLE 2

What does the formal observation process include in 2012-13?

Source: Modified from REACH Students Teacher Practice

Note: In 2012-13 administrators were required to conduct at least four formal observations for each non-tenured teacher and at least one formal observation for each

tenured teacher.

Pre-Observation Conference

A brief 15-20 minute

conference with a focus

on Domain 1 (Planning

and Preparation)

The teacher and

administrator decide

which lesson will be

evaluated

Post-Observation Conference

The teacher and

administrator discuss the

classroom observation

The teacher's self-reflection

is evaluated for Component

4A (Reflecting on Teaching

and Learning)

Ends with suggestions for

improving teacher practice

Classroom Observation

The administrator observes

teacher for about 45 minutes

Observation primarily

focuses on the components

in Domain 2 (Classroom

Environment) and Domain 3

(Instruction)

The administrator gathers

evidence and assigns ratings

teachers in their building. Ratings across classroom

observations are combined with test score gains to

give each teacher an o icial evaluation score.

In this chapter we examine teachers and adminis-

trators perceptions o the teacher practice component.

Our indings draw on both sur vey and interview data.

Survey data show the extent to which teachers and

principals across the district have positive or negative

view s ab out t he ob ser vations pr oces s. O vera ll, both

groups ind the process to be a useul means o helpingteachers improve their instructional practice. Teachers

appreciate the eedback they receive rom their evalu-

ators and believe the rating process is transparent.

Admini str ators thi nk the observation proc ess wi ll lead

to improvements in teaching and st udent learning.

Interview data provide insight into how the observa-

tion process supports instructional improvement. In

addition, it highlights teachers descriptions o how the

coordination o the evaluation process can undermine

the value o t he observations as an improvement tool.


12/44


10

The Observation ProcessSupports Proessional GrowthAdmini str ator s and te acher s ex press ed positive v iews

o the teacher practice components potential to sup-

port teacher growth and proessional development. On

the survey, 76 percent o teachers said the evaluationprocess at their school encourages their proessional

growth. Similarly, 76 percent o admin istrators

reported believing that the observation process would

result in instructional improvement at their school,

and 82 percent reported noticeable improvements in

hal or more o the teachers they had observed over

the school year (see Figure 1).

In interviews, teachers identiied three ways in

whic h the obser vation comp onent suppor ts t eacher

learning. First, they remarked that the Framework

rubric sets clear expectations about quality instruc-

tion. As one teacher succinctly put it: The observation

rubric describes what really good teaching looks like. It

gives me a clear descr iption o what teaching look s like

at each level.Responses on surveys indicate that ma ny

teachers and administrators agree w ith this sentiment:

75 percent o teachers and 91 percent o administra-

tors reported that the Framework provides a common

deinition o high-quality and eective teaching. Clear

descriptions o quality instr uction help teachers tran-

scend their own individual opinions about teaching and

begin to compare their practice to others. One teacherexplained:

You get into your own practices and orm

your habits and methods. But because

everyone is working within the REACH

system, you can start to see where you are

in the system. Everyone is breathing the

system language. I they all are refecting

the same language, you have to think about

others and all the other teachers.

Because it creates explicit and shared expectations

o quality inst ruction, teachers and administrators

commented that the rubric also provides clear g uidance

about what teachers need to address in order to improve

their practice:

I always thought there needed to be higher

standards in teaching, and I think the

observation rubric has made the standards

higher. [Beore] it was up to the principals

discretion o how he or she elt. [Now] its

clearer about what that means, how to grow,

how to improve. CPS Teacher

[In post-conerences] instead o just saying,

You got a 3 here and a 2 here, we can say,

What is the dierence between a basic and

procient [rating]? I didnt see this, I didnt

see this. And it was a really clear thing,

Start doing that, or, Stop doing something

else. CPS Principal

Admini str ator s were vir tua lly una nimous on th is

point: 96 percent responded on the winter survey that

the Framework helps them identiy areas where teach-

ers can improve.

Second, teachers reported that the teacher prac-

tice component has potential to improve instruction

Pe

rcentofAdministrators

70

60

50

100

90

80

40

20

10

0

30

Have incorporated

your feedback into

their teaching?

(n=622)

Have made noticeable

improvments over

this year?

(n=621)

Source: Spring Administrator Survey, May 2013

Note: Percentages may not add up to 100 due to rounding.

All Most About Half A Few or None

FIGURE 1

Most administrators report at least half of their

teachers have incorporated feedback and improved

Of the teachers you have observed this year, how many...

10%

63%

19%

8%

5%

48%

29%

17%


13/44


because it creates opportunities to discuss teaching

with ad min istrators a nd colleagu es. I n pa rticular, t he

pre- and post-conerences were a way o getting needed

eedback and support:

I love being able to rene what I do andtalk about it with somebody. So the idea

is that I get to sit down every month or so

and say this isnt really working or me and

my administrator will nd something that

can help me. That is really benecial.

CPS Teacher

I think that it is nice to have someone in

the classroom requently to really see

how youre doing and what youre doing

and give you eedback in a way that is not

really an attack. Its more like a positive,

constructive criticism on dierent aspects

o teaching. CPS Teacher

These comments highlight that teachers value

eedback on their instruction. They also show

that conversations with administrators tend to be

respectul a nd supportive. In act, only 6 percent

o teachers on the winter survey said eedback was

delivered in a hurt ul manner. Across the district, 82

percent o teachers indicated they have proessional

conversations with their administrators ocused

on instruction, 89 percent said their evaluator

supports their growth, and 76 percent reported

that their evaluators eedback was useul. Among

administrators, 94 percent thought the Framework

has improved the quality o their conversations with

teachers about instruction.

Finally, teachers also noted the conversations

helped them intentionally re lect on their own class-

room practice. I think its good to see what you did

and how you can improve,one teacher said. I cant

see mysel teach, and I love to hear how I can improve.

By creating opportunities to examine their own prac-

tice, the observation process helps teachers identiy

their strengths and weaknesses, as well as prioritize

areas on which to ocus their improvement eorts.

For some, the relection habit carries outside o the

ormal observation structure to their teaching more

generally: [The Framework] causes us to be more c on-

scious o our planning and the words coming out o our

mouth. It causes us to really look at what we are doing in

our classrooms.

Admini str ator s ag reed: 92 perc ent o principal andassistant principal survey respondents thought the

Framework encourages teachers in their school to

relect on their instructional practice (see Figure 2).

On the winter survey, 81 percent o teachers said it helps

them identiy a reas where their teaching is strong, and

82 percent said it helps them identiy areas where they

can improve.

Most Teachers Believe Administrator

Ratings were Accurate and Fair

Teachers were generally positive about the accuracy

o the ratings they received rom school administra-

tors. On the spring survey, 87 percent o teachers said

their evaluator was air and unbiased, and 88 percent

said they were able to assess their instruction accu-

rately (see Figure 3). On the winter sur vey, 72 percent

o teachers said their ratings were about the same or

higher than they thought they should have been.

One reason teachers were positive about their

ratings is they believe the speciicity o the Framework

helps makes ratings more concrete:

I like that they [the Framework] actually

speciy what it is that we are being

evaluated on, versus the old system where

your principal essentially gave you a rating

and some comments about what youve

been doing. With this [system], youre either

doing this or youre not. I youre not, then

youre not meeting [the standards]. I you

are, then youre procient.

Anot her re ason teachers eel rati ngs tend to b e

generally objective is because administrators have to

collect and present evidence about what they specii-

cally saw during observations. [The Framework] holds

a lot o accountability,one teacher said. Not only or

the teachers but also or the administrator; they have to

prove e verything t hey ve ound.


14/44


12

FIGURE 2

Nearly all administrators report the CPS Framework is useful for instructional improvement

The CPS Framework...

PercentofAdministrators

2%

5%

53%

40%

70

60

50

100

90

80

40

20

10

0

30

Is a useful tool for

identifying teachereectivenessin this school

(n=622)

Is a useful tool for

providing targetedsupport for teachers

(n=623)

Encourages

teachers in thisschool to reflect ontheir instructional

practice

(n=622)

Has provided a

definition ofeective teaching

in this school

(n=623)

Has improved the

quality of myconversations with

teachers in this

school about

instruction

(n=620)

2%

5%

50%

44%

2%

7%

58%

33%

2%

6%

54%

38%

2%

5%

54%

38%

Strongly Agree Agree Disagree Strongly Disagree



FIGURE 3

Most teachers believe their evaluator has the capacity to assess instruction fairly and accurately

My evaluator...

PercentofTeachers

5%

8%

24%

63%

70

60

50

100

90

80

40

20

10

0

30

Is fair andunbiased

(n=13,600)

Is able toaccurately assess

teachers instruction

(n=13,622)

Knows my strengthsand weaknesses

(n=13,624)

Knows whatsgoing on in my

classsroom

(n=13,627)

6%

13%

30%

51%

5%

10%

30%

54%

3%

9%

29%

59%

54%

To a Great Extent Some A Little Not At All

Source: Spring Teacher Survey MVMS, April 2013



15/44


While havi ng positive p erceptions o their rating,

some teachers thought using multiple raters could

improve the reliability o the ratings and how they are

used in personnel decisions. One concern is that being

observed by only one evaluator may lead to inaccurate

ratings. Recent research seems to validate this view,inding that multiple observers produce more reliable

ratings. 25I think it would be s omething to think about

having each observation done by a second person,one

teacher stated. Yes, ratings are evidence based, but

[evaluators] do ma ke interpretations based on previous

knowledge o you. So I think that ties into how they view

what you are doing and i you are doing it well.

In addition, some teachers worry administrators

may use observation ratings to remove or deny tenure

to sta they do not like. Thus, the ratings one receivesmay have little to do with what actually happens dur-

ing an observation. I just eel like it shouldnt be the

administration who is doing the observation on you,one

teacher said. Because the bottom line is that i they dont

want to keep you in the school, they are not going to. They

are going to give you a bad observation rating.Some

cities have addressed this problem by incorporating

governance structures that support personnel systems.

These systems ty pically include the use o expert men-

tor teachers as evaluators and coaches, as well as review

structures or personnel decisions that involve teachers

and administrators in the decision-making process.26

Administrators Dual RoleCan Undermine ProessionalLearning BenetsREACHs reliance on administrators both to ocially

evaluate teacher practice and to provide instructional

coaching may undermine the learning potential o the

observation process. Because observation ratings have

such a big impact on summative evaluation scores, teach-

ers are highly motivated to demonstrate their proessional

competence when they are observed. Since the admin-

istrator giving ocial ratings is simultaneously provid-

ing instructional support, however, teachers are orced

to weigh the costs and benets o using the observation

process as an opportunity to share their instructional

weaknesses. For some teachers, the risk o receiving a

poor rating is too great. As one teacher explained:

Because there is such an emphasis placed

on assessing the quality o teachers, there is

no incentive or teachers to admit insecurity

or talk about areas in which he or she strug-

gles. I elt like I had to mask the things that I

didnt do as well and try to explain why theydidnt go well because, at the end o the day,

Im being rated. So there is more o an incen-

tive to present mysel avorably than to have

an honest discussion about instruction.

Several teachers across our interviews schools

described instances when they perceived that attempts

to get support or addressing weaknesses led to nega-

tive consequences on their evaluation. For example, one

teacher said he was very honest at the start o the yearabout his practice by highlighting or his evaluator things

in my daily teaching that I need to strive to x.Ater being

inormally warned that his evaluator would pay more

attention to those areas, he elt as though the eva luator:

ended up putting a laser ocus on the

things that I do want to x, but are hard to

x. Instead o being rewarded or being

sel-aware and honest about improvements,

I eel like I m actually being penalized.

Anot her teacher recount ed goi ng to her pr incipa l

or help regarding classroom management issues in

one class. Instead o receiving support, she elt the

request led to her receiving a low observation rating:

My principal joked around and said hell do my

next observation in that classroom [in which I

was struggling]. It was a joke, and then he

actually did it. When I said I really dont know

i it is appropriate or me to be judged based

on that classroom, when there are so many

other classrooms and grades that I teach

where Ive already been observed, I was scold-

ed and told that I am essentially saying that I

cant do my jobIt makes me eel like I cant

even come to my own administrator or help,

because that inormation was essentially used

against me in the observation process.


16/44


14

These two comments highlight the potential risks

involved in asking an evaluator or support. I teach-

ers present a realistic view o their teaching, they may

be rated as less skilled compared to others who put on

a perormanceduring a scheduled observation. It is

important to keep in mind that we do not have evidence

about how widespread instances like the ones above

are. Nonetheless, these cautions show how the learning

opportunities created by the observation process could,

in the long run, be undermined when the evaluators

giving ratings are also primary instructional coaches.


17/44

Chapter 2 | The Use o Student Growth in Evaluation

The Use o Student Growthin Evaluation

CHAPTER 2

Prior to REACH, teachers in Chicago were not held or-

mally accountable or the perorma nce o their students.

The use o student growth to measu re teacher peror-

mance breaks new ground. A teachers student growth

score summarizes t he change in his or her students

standardized test scores between two time periods:

under REACH, the beginning and end o a school year.27

PERA requires that student growth be a signicant

actor in teachers evaluations, though CPS and the

CTU agreed to phase in t his requirement so that studentgrowth accounted or no more tha n 25 percent o a teach-

ers evaluation score in the initial yea r. The weight given

to student growth in a teachers nal evaluation varies

according to the subject and grade level o the teacher

(see What Goes Into a Teachers Evaluation Score?on

p. 7). Student growth is ca lculated di erently depending

on the assessment that is used. These assessments also

var y by g rade and subject , but they can i nclude:

A gain score on district-developed Perormance

Tasks, which are w ritten or hands-on assessments

speciically designed or the grade and subject o the

course and are most oten scored by the teacher

A value-added score on the NWEA MAP, an adaptive,

computer-based test administered to students in

grades 3-8 in reading and math

An expected gains score on the subject area

EXPLORE, PLAN, or ACT ( EPAS) a ssessments

administered to students in grades 9-11 in English,

reading, math, and science28

A measure o average schoolwide literacy growth

rom either the NW EA MAP or the EPAS

The Student Growth Component o REACH box

on page 20 urther describes the measures o st udent

growth used in RE ACH.

As s et orth by PE RA , st udent g row th i s inc orpo -

rated strictly or evaluation purposes. However, CPS

has been clear that they expect REACH to positively

aect teacher development and student learning. I the

student growth component is to be useul beyond teach-

ers evaluations, it must provide teachers with inorma-

tion that can inorm their instruction. A student growth

score alone does not provide teachers with inormation

that is timely or detailed enough to guide improve-

ments in their instructional practice; it is one number

that summarizes changes in test scores across a group

o students over a given period o time. In contrast,

students perormance on the individual assessmentsused to calculate student growth might inorm teach-

ers instruction by providing them with inormation on

their students skills or level o understanding.

In this chapter, we describe teachers responses to

the use o student growth in t heir evaluations, as well a s

how useul teachers ound the assessments or their in-

struction. We ind apprehension among teachers about

the incorporation o student growth metrics into their

evaluation. Teachers were generally positive about the

potential instructional value o the assessments used

to measure st udent growth, though the perceived use-

ulness varied considerably by the assessment.

Teachers Are ApprehensiveAbout the Use o StudentGrowth in Their EvaluationGiven that student growth is a new addition to teach-

ers evaluation, it is not surprising that many teachers

expressed concerns over its use in measuring teacher

perormance and in personnel decisions, or that many

were misi nor med or con us ed about how student

growth actors into their evaluation. Additionally,

some teachers raised concerns about the potential

or bias when applying the student growth measures

across dierent classroom contexts.

On our survey, 57 percent o teachers said that they

believe or strongly believe that the REACH system

relies too heavily on standardized tests (see Figure 4).

Anot her 30 percent sa id th at they somew hat b elieve


18/44


16

this, while only 13 percent o teachers said that they do

not believe that REACH relies too heavily on standard-

ized tests. We asked teachers an open-ended question

about what they ound most problematic about the

REACH system. Nearly one-third o the 552 teachers

who r esponded t o this quest ion ident ii ed the studentgrowth component and the assessments used to mea-

sure student growth, making t hese the most requently

cited problematic aspects o REACH. 29 While some o

these teachers maintained that test scores should never

be used in teachers evaluations, others identiied more

speciic concerns. These concerns included the narrow

representation o student learning that is measured by

standardized tests, t he numerous inluences on student

perormance that are outside o a teachers control,

and an increase in t he already heavy testing burden onteachers and students.30

Teachers responses to our interview and an open-

ended survey item revealed that many o them were

misinormed or unclear on how much student growth

contributes to their summative evaluation. For exam-

ple, one teacher wrote, I am concerned about my eort

as a teacher completely relying on the test scores o my

students.In act, student growth does not account or

more than 25 percent o any teachers evaluation this

year, and student growth w ill not account or more than

30 percent once REACH is ully implemented; there-

ore, no teachers evaluation will completely rely on test

scores. Some teachers urther att ributed the incorpora-tion o student growth to the district, rather than to the

state law. As we show in the next chapter, most teachers

reported receiving inormation about REACH rom

their school administration. Yet only 45 percent o

principals a nd assistant principals reported having a

strong or very strong understanding o how student

growth actors into a teachers summative rating, so

it is not surprising that teachers are also unclear.

Several teachers expressed concerns that measures

o student growth are unair to teachers in more chal-lenging schools because student growth, and thereore

a teachers evaluation score, is related to the supports

that students may or may not receive outside o the

classroom. One teacher explained this concern:

Im not going to want to work in a [strug-

gling] school i my evaluation is tied to test

scores, because there are things that I cant

control. I cant stop gang violence. I cant

stop poverty. I cant stop the parents who

dont care i their kids go to school. I think

the part that I nd unair is that so much o

what goes on in these kids lives is aecting

their academics, and those are things that a

teacher cannot possibly control.

Related to the issue o airness, many teachers

expressed apprehension over how the student growth

measures would be used by the district in part icular

that they would be used to ire teachers or to institute

merit pay. For example, one teacher explained that she

had grave concernsthat her students perormance

could negatively impact her job security, in part because

there are so many other actors outside o the classroom

that inluence student growth.

Two groups o teachersspecial education

teachers and non-core subject teacherswere

particularly critical o the student growth component.

Special education teachers raised concerns that

PercentofTeachers

70

60

50

100

90

80

40

20

10

0

30

Strongly Believe Believe

Somewhat Believe Do Not Believe

(n=731)

13%

30%

24%

33%

Source: Winter Teacher Survey, January 2013


FIGURE 4

Most teachers believe or strongly believe REACH

relies too heavily on standardized tests

Please indicate the extent to which you believe that

REACH, overall, relies too heavily on standardized tests.


19/44


the REACH Perormance Tasks, NWE A MAP, and

EPAS assessments were inappropriate measures o

their instruction and o their students learning. One

special education teacher explained: The grade level

REACH Pe rorma nce Tasks we re nearly impossible or

my special education students, and it will be diicult toshow improvement or many students who are our and

ive grade level s behind.This teachers concern was

echoed by many special education teachers who believed

that holding their studentsand, thereore, their own

evaluationto the same standard as regular education

students and teachers was unair. Many teachers were

unclear on what accommodations could be provided

or their special education students as they took the

assessments, which were the same assessments that

were g iven to reg ula r educat ion st udents.

31

Some non-core subject teachers (e.g., art, music, and

physical education) were troubled by the incorporation

o schoolwide literacy growth into their eva luation.

These teachers disliked being held accountable or the

work o other teachers a nd or a cont ent a rea that they

were not ne cess ar ily prepa red t o te ach. For exa mple, a

high school art teacher explained his eelings about the

schoolwide literacy measure:

The comment has been made that I will get

judged on reading scores because we are all

teachers o literacy, and theres a part o me

that agrees with that. But...there is no part o

my certication or training that says I need

to learn how to teach a student how to read.

Teachers Found Beginning-o-YearREACH Perormance Tasks UseulThe REACH Perormance Tasks were developed by

teachers and district specialists as Type III assessments

(see The Student Growth Component of REACHon

p. 20). As dened by PER A, Type III assessments ar e

rigorous, aligned to the courses curriculum, and mea-

sure student learning in that course. Perormance Tasks

are a written or hands-on demonstration o mastery,

or progress towards mastery, o a particular skill or

standard,32 which make them very dierent rom

traditional multiple-choice assessments. The primar y

purpose o the RE ACH Perormance Tasks is to provide

a measure o student understanding at the beginning

and end o the school year so that a growth score can

be calculated and incorporated into teachers evalua-

tions. In the best ca se, however, the beginn ing-o-year

Perormance Tasks would also provide teachers with

inormation that is useul or their instruction, suchas inormation about their students skills or about

the districts expect ations or what content should be

covered in their class.

Amon g teacher s who admini ster ed a beg inn ing- o-

year REACH Perormance Task, 70 percent reported

that it was somewhat or very useul or their instruction

(see Figure 5). In inter view s, teachers r epor ted using

the Perormance Tasks as an indication o what mate-

rial they needed to cover. Moreover, teachers seemed

to appreciate the more comprehensive set o skills thatstudents could demonstrate on the Perormance Tasks:

72 percent agreed that the tasks provided inormation

that is not measured on traditional multiple-choice

assessments.

Wh ile the Peror mance Ta sks prov ided t eachers

with in sight into what materia l they ne eded to c over,

ew teachers used the Perormance Tasks as measures

o student understanding. Two-thirds o teachers

(67 percent) agreed that the Perormance Tasks were

rigorous assessments o st udent learning, but they may

have been too rigorous; nearly the same proportion

(66 percent) indicated that the tasks were too chal-

lenging or beginning-o-year assessments. Because

the Perormance Tasks oten covered material that

the students had not yet been exposed to, they did not

provide a measure o students understanding o that

material. Rather than test students prior knowledge,

the Perormance Tasks assessed students on content

that they had not been taught. One teacher explained

why t he le vel o cha llenge is par ticu lar ly a problem or

a beg inning-o-year assessment:

[my students] were really upset by it. Not

only because it was something they had

never seen beore, but they didnt know

me, so it was kind o like they didnt know

me and I was giving them something and

challenging them in a way that it was unair

or them and it made me eel really bad.


20/44


18

This teacher explained that once she has gotten to

know her students, she can motivate them to persevere

through a chal lenging assessment. But without the time

to build those relationships, it was diicult or her to

help her students overcome their rustration.

There was airly widespread conusion amongteachers about the administration o the Perormance

Tasks. Just 41 percent o teachers who had adminis-

tered a beg inning-o-year Perormance Task indicated

that they were clear on how the tasks should be scored.

Over one-third (36 percent) o teachers indicated they

did not have adequate time to score the tasks, and

one-third (35 percent) indicated they had diiculty

recording the scores on the districts internal site. Just

under hal (43 percent) o teachers indicated that they

were not at all cle ar on what ac commo dations c ouldbe made or students with IEPs who were taking a

Perormance Task.

Apart rom t heir instruc tional value, severa l

teachers raised concerns about how easy it would be to

game the scoring o the Perormance Tasks. Teachers

score their own students Perormance Tasks at both

the beginning and end o the year. In an interview and

on an open-ended survey item, teachers noted that i

they wanted to max imize their student growth score,

they could simply give all students a low score on the

beginning-o-year task and a higher score at the end o

the year. While we have no measures o how requent-

lyi at allthis practice occurred, it has the potential

to undermine how teachers and administrators per-

ceive the accuracy o the eva luation ratings.

NWEA MAP Provided Timelyand Useul DataThe NWE A MAP is a series o computer-based,

adaptive assessments administered to CPS students

in grades 3-8 at the beginning and end o the school

year.33 Seventy-eight percent o teachers who admin-

istered a beginning-o-year NW EA MAP assessment

ound it somewhat or very useul or their instruction

(see Figure 5) and a simila r prop ortion (75 percent)

agreed that the NW EA MAP helped them to target

their instruction to meet students individual needs.

PercentofTeachers

70

60

50

100

90

80

40

20

10

0

30

Performance

Tasks

(n=13,253)

EPAS

(n=3,225)

NWEA MAP

(n=6,316)

Source: Spring Teacher Survey MVMS, April 2013

Note: Responses only include teachers who conducted the assessment(s) in the

fall of 2012. Only high school teachers in core subjects administered EPAS. Only

elementary school teachers in grades 3-8 reading and math administered NWEA

MAP. All elementary school teachers and a subset of high school teachers in core

subjects administered Performance Tasks. Percentages may not add up to 100

due to rounding.

Very Useful Somewhat Useful

A Little Useful Not Useful

FIGURE 5

Teachers report assessments vary in instructional

usefulness

How useful is the following for your instruction?

14%

17%

32%

38%

8%

14%

34%

44%

22%

29%

34%

16%

The rigor o the NWEA MA P assessments and the

timeliness with which teachers receive their students

results may help to explain the NWE A MAPs instruc-

tional value. Eighty-ive percent o teachers who had

administered the beginning-o-year NW EA MAP

ound it to be a rigorous assessment o student learning.

Addit ionally, t he major ity o teachers a greed t hat the

results provided by the NW EA MAP are both timely (92

percent) and easy to understand and use (72 percent).

Wh ile the computeriz ed natu re o the NW EA M AP

assessments likely contributed to their instr uctional

useulness, it also created problems or some teach-

ers. Over two-thirds o teachers who had administered

the beginning-o-year NWE A MAP reported that they

experienced technical diiculties, such as issues with

computer hardware or internet access. One teacher

explained how technical problems can aect student

perormance:


21/44


At our school, our technology isnt up-to-

date. The computers themselves are about

nine or 10 years old.When everybody was

taking the test at once, that was an issue

because our routers couldnt handle the

amount o trac. So the internet would goout. I think that really skewed our test results

because students, especially on reading,

would have to read the story and then go

back to it and then they were stuck and they

would have to go back. The students wont

reread what they read, so they might orget

a part, and then theyre asked questions.

How to use the NWE A MAP with particular popula-

tions o students was again a concern or teachers: 30percent were not at all clear on what accommodations

were acceptable or special edu cation st udents and 39

percent were not at all clear on whether ELL students

should take the NWE A MAP.

EPAS Results Not Timely or DetailedHigh school teachers were less positive about the value

o the beginning-o-year EPAS assessments or their

instruction than elementary teachers were about the

NWE A MAP a ssessments. EPAS assessments are given

in grades 9-11, as par t o ACTs testing system. While 71

percent o teachers who had administered a beginning-

o-year EPAS assessment agreed that the test was a rig-

orous assessment o student learning, only 50 percent

o those teachers reported that it was somewhat useul

or very useul or their instruction (see Figure 5).

One issue limiting the instructional value o the

beginning-o-year EPAS assessments is the timeliness

with wh ich teachers r ecei ve their resu lts. Unli ke t he

computer-based NWE A MAP that provides teachers

with result s im medi ately ollowi ng the a ssessment ,

teachers do not receive their students EPAS scores or

several months. Just 50 percent o teachers who admin-

istered a beginning-o-year EPAS assessment indicated

that they had received their students results in a timely

manner. In our interviews, we heard why this delay is

problematic or teachers:

We didnt get the results back until basically

almost January, so its kind o like the data

is dead...[it] refected what they knew three

months ago. I I had gotten an item analysis,

that would have been more helpul. But I just

got a raw score so I know that they scored a14.I know I want to improve that score, but

I dont know why theyre getting that score.

Moreover, as the teacher above explained, the results

o the EPAS exams are not detailed enough to guide

teachers instruction.34 Teachers receive their students

subject scores, but not an item analysis. Just 44 percent

o teachers indicated that the beginning-o-year EPAS

helped them to pinpoint individual students strengths

and weaknesses.Since high schools had administered EPAS as paper

and pencil exams or a number o years beore the

implementation o REACH, their administration caused

less widespread conusion among teachers than the

NWE A MAP or REACH Perormance Tasks. However,

the issue o how to use the tests with special education

and ELL students remained: 25 percent o teachers

indicated that they were not at all clear on what accom-

modations were acceptable or students with IEPs and

35 percent were unclear on whether their ELL students

should take the assessment.

As thi s cha pter and the one t hat precedes it show, the

implementation o the observation process and student

growth component o REACH required substantial

eort rom teachers and administrators. In the next

chapter, we explore the challenges and successes o

training and communication to support this eort.


22/44


20

The Student Growth Component o REACH

Illinois Perormance Evaluation Reorm Act (PERA)

denes three dierent assessment types:

Type I assessments can be the typical multiple-

choice standardized assessment that measures

a certain group or subset o students in the same

manner with the same potential assessment items,

is scored by a non-district entity, and is adminis-

tered either state-wide or beyond Illinois.

Type II assessments can be any assessment

developed or adopted and approved or use by the

school district and used on a district-wide basis

by all teachers in a given grade or subject area.

Type III assessments are rigorous, aligned to

the courses curriculum, and are determined by

the teacher and qualied evaluator to measure

student learning in that course.D

PERA stipulates that all teachers must be evalu-

ated using at least one Type I or Type II assessment

and at least one Type III assessment.E To meet this

requirement, CPS has identied two dierent types o

student assessments to be used as part o REACH:

REACH Perormance TasksAs its Type III assessment, CPS utilizes REACH

Perormance Tasks, which were administered in

the all and the spring and are intended to measure

change in student mastery over one or two skills

or standards. The REACH Perormance Tasks were

developed by over 150 teachers organized into teams

aided by content area specialists rom central oce.

These teams developed over 90 Perormance Tasks

that covered all elementary teachers, including those

teaching in areas such as art, music, physical educa-

tion, and library studies that are not traditionally

covered by standardized tests and a subset o teach-

ers in high school core courses. Each Perormance

Task all/spring pair took approximately 40 hours to

drat, revise, and pilot.

Value-Added and Expected Gains Measures

For its Type I assessment in elementary schools, CPS

has chosen to compute teachers value-added score

on the math and reading NWEA MAP. A value-added

score rom the all to spring administrations o the

NWEA MAP will be computed or teachers who teach

grades three through eight reading or math. All other

elementary school teachers will receive a schoolwide

literacy growth score.

For its Type I assessment or high school teachers,

CPS is exploring using the EPAS suite o tests

(EXPLORE, PLAN, and ACT) to measure expected

student gains. In 2012-13, the EPAS assessments

were administered without stakes. EXPLORE was

administered twice to ninth-graders, PLAN twice to

tenth -graders, and ACT twice to eleventh-graders.

While these scores will not count towards teachers

evaluation this year, the data will be used to develop

an expected gains metric or possible use in the

2013-14 school year.


23/44

Chapter 3 | Training and Communication

Training and Communication

CHAPTER 3

As with the implement ation o a ny major polic y in itia-

tive, REACH required ex tensive communication and

training eorts at both the district and school levels. In

Chicago, almost 1,200 administrators a nd over 20,000

teachers needed to be inormed about and trained on

the new system in this irst year o implementation.

In this chapter, we describe teacher and administrator

experiences with trai ning and communication. We draw

upon our surveys and interviews o teachers and admin-

istrators to understand how well-inormed and preparedparticipants elt in this rst year o i mplementation, and

to explore what area s they elt still needed improvement.

We nd that , while admi nistrators re ceived ex tensive

training, training and inormation or teachers varied

widely both acro ss and within schools. Fi nally, teachers

and administrators alike expressed a need or transpar-

ency not only about how nal summative scores would be

calculated but also about how teacher evaluation would

ultimately be utiliz ed in personnel decisions.

Administrators Felt Preparedto Conduct Observations andAssign RatingsMore than 80 percent o administrators reported

their proiciency as strong or very strong in recording

and aligning evidence and determining observation

ratings. Administrators received extensive training

in these a reas. Prior to conducting any observations,

administrators had to complete an online certiication

process that included video-based scoring practice and

an assessment o their rating accuracy. On average,

administrators reported spending over 30 hours on

this certiication process. Administrators who did

not pass the a ssessment portion o this certiication

ater two attempts were required to attend additional

in-person training. Adm inistrators who did not pass

the assessment portion ater our attempts did not

conduct observations. As o November 2012, almost90 percent o CPS administrators had been certi ied.35

Beyond certiication, the district required admin-

istrators to attend our hal-day, oundational REACH

proessional development sessions throughout the year.

These sessions included content on the teacher practice

component, evidence alignment, rating calibration, and

evidence-based conversations. On our administrator

survey, about 70 percent o administrators said they

ound district-provided proessional development on

REACH helpul or very helpul.

In interviews, administrators reported conducting

joint obser vations wit h another pr actitioner and dis -

cussing the evidence and ratings they assigned was most

relevant to helping them eel prepared. Administrators

oten relied on their networks Instructional Eec-

tiveness Specialist or this on-the-job training and were

positive about the individualized coaching they received

(see Instructional Effectiveness Specialists box). Over

Instructional Eectiveness Specialists

Specialists conduct joint classroom observations

and calibration sessions with administrators in

their schools. Specialist ratings are not directly

incorporated into any teachers evaluation rating.

Instead their purpose is to work with administrators

on their ability to assign unbiased ratings based only

on the evidence they collected during observations.

In 2012-13 there were 18 specialists, approximately

one or every school network (although some

were not hired by the time school started in the all).

Individual specialists backgrounds, experiences and

capacity varied, but all were certied and trained

to ensure evaluator quality and inter-rater reliability

as well as identiy evaluators needs in conducting

observations.


24/44


22

80 percent o administrators ound their conversations

with their netw ork speci alist help ul or ver y help ul

in the areas o evidence collection, alignment, and

assignment o ratings. In interviews, principals stated

they ound conducting joint observations with their

specialist helpul because they could ocus on theirindividual needs as observers.

Principals also relied heavily on their own admin-

istrative teams or training and ca libration. In our

interviews, principals reported spending signiicant

time training, jointly observing, comparing evidence,

and discussing ratings with their assistant principals at

the beginning o the year. One principal and assistant

principal team even planned to conduct all o their

observations together because they elt it would

maximize t heir learning and ensure calibration:

We did them together because we wanted

to coordinate and do what we called, max

calibrate. We wanted to make sure we were

on the same page on what we classied

this is 2A, and then this is 3Bso we had

those discussions, which took a long time.

We just thought that would help us to grow

and evolve.

Principals Want AdditionalTraining on Coaching TeachersWhen asked what resource s or tra ini ng they needed or

observations and conerences, administrators most re-

quently identiied help with acilitating teacher growth

and development. Ninety-our percent reported proes-

sional development on providing useul eedback and 97

percent reported coaching teachers as a high or medium

training priority (see Figure 6).

In interviews, principals identiied speciic issues

with provid ing eedb ack t o te acher s ba sed on obser va-

tions. One principal described having diiculty priori-

tizing areas or improvement with struggling teachers:

Theres 15 things they need to get better at, and so all

15 o them are im porta nt, whe re do I begin?Anot her

administrator talked about struggling to ind the

best way to reach each individual teacher:

An area I still struggle with is... when some-

bodys doing something wonderully, how

to be a good thought partner with making it

better... Just knowing that people want di-

erent things, some people want more direct,

like, This is what you should do, and some

people that turns o immediately. So its just

nding, with each individual, what is going

to be the right piece to reach him or her.

A ew princ ipal s des cribed str ugg ling w ith provi ding

useul and thoughtul eedback to teachers in the ti me

they had allotted: How do you do it well, and have

really thoughtul conversationsI just dont know how

you would get any r eal inormat ion and push somebody

in 25 minutes?Another principal elt that the post-

conerence only allowed her time enough to report to

the teacher what I thought she needed to work onand

not to provide speciic steps on how to improve.

PercentofAdministrators

70

60

50

100

90

80

40

20

10

0

30

Providingprofessional

development

to teachers toaddress specific

needs

(n=612)

Coachingteachers to

improve their

practice

(n=609)

Providinguseful

feedback

to teachers

(n=612)



High Priority Medium Priority

Low Priority Not a Priority

FIGURE 6

Most administrators report coaching and feedback

as high priorities for their own training

How high a priority are the following topics for your ownprofessional development?

70%

25%

4%

1%

4%

75%

19%

1%

79%

18%

2%

1%


25/44

Chapter 3 | Training and Communication

Specialists also said their administrators were un-

easy about conerencing with t heir teachers. Specialists

stated some administrators were unsure o the purpose

o the pre-observation conerence, while other adminis-

trations told them they were eeling uneasy about

leading conerences where they have to tell teachersthey received lower ratings than in t he past.

Training and Communication orTeachers Varied WidelyFor teachers, the requency and quality o REACH

trainin

Documents

Teacher Evaluation in Practice: Implementing Chicago's REACH Students