18 | FocalPoint

photos by Joshua Dunn

FocalPoint | 19

CICS Executive Director Beth Purvis and board member Jerry Jenkins reflect

on CICS’ decision to implement the Northwest Evaluation Assessment

Measures of Academic Progress accountability system—the rationale

behind the move, mistakes made along the way, and, ultimately, lessons learned.

Talking Points

interview by Katie Nugent

TalkingPoints

20 | FocalPoint

Why did you undertake the accountability system at CICS?

Beth Purvis: Our mission is college-preparatory education. In order to adequately determine whether or not we are meeting our mission, we needed a more accurate and timely analysis of student performance, and the state exams don’t tell us about individual student growth. And, they are not timely in their responsiveness.

At the time, were there other educational organizations doing this? Did you have contact with them during the development of CICS’ system?

BP: Yes. During the strategic planning process the Board undertook during the 2004-2005 school year, one of the things we did was an analysis of the high-performing urban school districts. What we found were two things. There are some examples of high-performing urban school districts that have accountability programs. Charlotte-Mecklenburg is an example of one. And then there are groups of charter schools and some of the smaller individual single-site charter schools around the country that are in urban settings that are doing value-added analysis.

The two systems for elementary school that we looked at most closely, which were Scantron’s Performance Series and NWEA MAPs, are both being used in districts across the country, so we were able to look at their implementation and their outcomes and how they worked.

Jerry Jenkins: I would say that for the most part, especially the larger examples, people use accountability systems to manage their internal workforce. As far as I know, we are unique in using an accountability system to manage subcontractors. And so we use a contractual approach along with our accountability system to manage who we hire, who we fire, and who we extend our relationship with.

What were the few major changes to designing and implementing an accountability system?

JJ: First, one didn’t exist when we started.

So we had to determine what is was. Add to that the fact that most people start with not a growth-based system but an input-based system, or worse, a status-based system.

BP: Status-based is how people are doing in relationship to a curriculum or other people rather than how they are doing in relationship to themselves over time.

JJ: Ours is a growth-based system where we figure out if people are achieving appropriate individual growth levels from year to year. And we knew we wanted to do that from the beginning, and that that was not the common way of doing it.

BP: For me, the biggest challenges were designing an affordable system that did all the things we wanted them to do. So, we wanted a system that gave teachers the formative data they need to make immediate decisions about curriculum and instruction. We wanted a system that could give a building principal or director the information that he or she needs on everything from personnel to personal development to long-term curricular decisions.

We wanted something I could use to manage the relationship with the educational management organizations, so we could create a rubric that said, if you do well, we renew our contract, if you do medium, you’re on probation, and if you don’t do well, we won’t renew. And we wanted something the Board could use to determine how they were doing toward their strategic plan. The number one priority of their strategic plan is to improve student outcomes.

We needed to do it in a way that was affordable and sustainable long-term. And, we needed something that teachers would love, because if teachers don’t like it, they won’t implement it. And if they don’t implement it, the data is neither reliable nor valid.

The first system we used failed because the teachers didn’t like it and we didn’t put enough money into professional development.

Once we found the system, we had to deter-mine how much professional development was enough and affordable to make sure we were getting reliable and valid data.

How did the Board and staff work together to bring the project to fruition?

BP: I think the Board hired me partially to do this. So the Board had the vision when they hired me.

We needed to build staff capacity at the same time we built this accountability system. We could build a world-class accountability system, but if we didn’t have the capacity for the training, implementation and analysis of the data, it was a waste of time.

As the capacity of the staff has grown, as the accountability system has grown, the focus of the Board has changed dramatically from growth and finances to education first. And that is a very different place than we were in five or six years ago. Not because the Board didn’t care, but back then we didn’t have the data to talk about it.

JJ: Beth came at exactly the right time. She not only knew how to do it, but her views of how children should be educated matched our vision precisely.

We all knew we needed an accountability system, we all knew what we needed in sound bytes, but we needed to put a lot of flesh on the bones before we could implement it.

How has the implementation of the system affected the classroom environment?

BP: It has created a transparency for teachers in what is expected. Your performance will be measured in part by the percentage of students who exceed their growth targets. It’s clear. The teachers know what is expected of them. And they have the tools to understand why students aren’t meeting their targets. So they have very clear and transparent goals for each child, but they also have a road map for each child that says why a child met their targets and why they didn’t. So it gives transparency to the goals but it also gives a

TalkingPoints

FocalPoint | 21

tool to help a teacher understand how to reach those goals. Because of that, pro-fessional development is far more focused. Curriculum and instruction is far better aligned with where students currently perform and what the goal of performance is.

JJ: For example, we can give a computer adaptive test to someone who’s not doing well in algebra. This test may well tell us that that child can’t do fractions. So you could spend three days of intensive remedial work with the child, and that person will never get algebra unless you back up and make sure the child knows fractions. The accountability system gives teachers the tools to figure that out.

BP: One of the other things we look at—it’s not just did you or did you not meet your target—it’s the percent-age of the kids who met their target and what quartile did they fall in. You cannot meet your target as a teacher in terms of the percentage of kids who meet their targets unless you differentiate instruction. It basically means that I am going to teach kids where they are currently functioning. This is the big buzzword in education. It has its origins in a one-room

schoolhouse where you had to differentiate your instruction because you had eighth graders and third graders in the same room. We’ve lost that because of some national tests, tests that just look at status score, reward teachers if they move kids from just

below the proficiency line to just above the proficiency line. You could teach to the middle and actually look as if you’re a phenomenal teacher. This won’t let you do that because we can look and say “Your kids in the second and third quartile met their targets, but the kids in the bottom quartile and the top quartile did not.” Which means you did not differentiate your instruction enough to meet the needs of those varied learners.

JJ: The accountability system is not something we expected to be mature when it was born. One of the key features of our program now is looking at all four quartiles and making sure people are progressing in all four quartiles. That was not a feature in the year we first implemented it. And what we found was, schools were thinking of ways they could meet their targets by focusing all attention on a narrow band of kids. We had created a system that was exploitable. What we did was change the system, and we will continue to tweak the system so we can use the accountability system to manage how they commit resources to individual problems.

We all knew we needed an accountability system, we all knew what we needed in sound bytes, but we needed to put a lot of flesh on the bones before we could implement it.

TalkingPoints

22 | FocalPoint

What are the lessons learned for other educational organizations that may be thinking about creating a similar system?

BP: You want to be both mission-driven and mission-constrained. We’ve always been a college-prep organization and we have focused our curriculum on creating a college-prep environment—we have the climate, the culture, the curriculum to do that. But we needed an accountability system to determine whether our student learning, our product, was actually met by our mission. The other thing the accountability system allows us to do is to be more constrained by our mission; to say, “These things are interesting and important but they aren’t mission relevant.” We can’t be all things to all people. To me, the accountability system allows me to do both.

The thing I love more about our accountability system than anything else is the first people who get the data are the most important people in the group, and that’s the teachers. The second thing is, your number one expense should be professional development in relationship to the implementation of the accountability system. That’s where we blew

it the first time. We chose something we liked, not that the teachers liked, and we did not spend the money to train them to use the system.

JJ: The second part of that, which is part of our strategic plan, is that a teacher who perceives a system is usable only to evaluate him or her, is not going to move heaven and earth to implement it. They have to want to listen as much as we want to teach them.

BP: The last lesson learned is that the accountability system is only a tool, and it’s only as good as the decisions you make from the data that is out there. Having the data is only part of the solution. You have to make smart decisions based on what the data tells you.

JJ: You have to be totally agnostic, in the sense that you can never believe your view of what works is what we have to do. You have to try it, and if the system shows positive results, then use it, but you have to believe the numbers.

Once gathered, how is the data shared with all the constituents of CICS?

BP: Teachers all get the data immediately. They have been trained in how to login and pull up the data and they can actually ask for different reports on their own classroom of kids. Then, the EMOs do their own analyses that they share with the teachers and use as part of their professional development. We share our analyses with the EMOs twice a year, based on their midyear and end of year performance. The end of the year is a much more extensive report that shows how they did on each campus relative to their

contractual obligations. Then, there is a much longer report that breaks down those data.

JJ: In the end, we’re pass-fail. The only gravity we have is to take the campus away from somebody or to award them more campuses. In the end, we have to boil down data from quartiles within classrooms, spread across grade levels, spread across campuses, down to one number.

BP: That number is how we make our decision.

How important is this accountability system to CICS?

JJ: It goes to the very essence of the organization. If we were not able to have an accountability system, I don’t think we would keep the passion of the board members and the team going.

BP: People in urban ed spend a lot of time talking about culture and climate. And we agree about the importance of a structured and disciplined climate and a culture of high expectations. However, that should be the just minimum setting of the bar.

JJ: If we haven’t instilled those attitudes and cultures in the first few grades, it’s almost too late. It’s not wasting time to instill that early on, because it makes it easier to achieve those goals because we have a partner as opposed to someone who’s fighting us. We are teach-ing people how to do well in this educational system. Kids who do well, once they’ve flown out of our nest, are the kids who’ve built character before they’ve left. It’s our job to set them up with that character.

The thing I love more about our accountability system than anything else is the first people who get the data are the most important people in the group, and that’s the teachers.