ALBUQUERQUE PUBLIC SCHOOLS - aps.edu · easily comprehended by the lay public. ... students speaking a language other than English or the negative effects of poverty. While the intricacies

ALBUQUERQUE PUBLIC SCHOOLS

INSTRUCTIONAL AUDIT, FALL 2012

SUMMARY AND KEY FINDINGS

Russ Romans Tom Genné January, 2013

Research, Deployment & Accountability

RDA/rr&tg/ 1.30.2013 Instructional Audit-Key Findings 2

Instructional Audit – Summary

The Obama administration inherited No Child Left Behind with its imminent failure looming as the nation

faced the statistical impossibility of achieving its primary goal. Obama’s Secretary of Education, Arne

Duncan, encouraged and urged states to request waivers from the law and rapidly granted them. In this

way NCLB, while continuing to play out in very few states, would eventually expire quietly and unnoticed in

the dustbin of well-intentioned but bad ideas.

New Mexico, under the gubernatorial leadership of Susanna Martinez, with expert advice from her

educational leadership team headed by Hanna Skandera, also applied for and received a waiver from the

feds and instituted a replacement accountability system for the school districts of New Mexico. The new

system was considerably different from NCLB with its impossibly challenging Adequate Yearly Progress

targets (“a rising tide lifts all boats”). The new system grades schools on a traditional A-F grading system,

easily comprehended by the lay public. Some of the most visible differences between the two systems are

identified below.

NCLB School Grades

One year of SBA results 3 years’ worth of data

Static picture Measures student growth over time

Pass/fail system (37 ways to fail with only 1 way to pass)

Understand where opportunities for improvement exist

“All or nothing” credit Partial credit given for all indicators

The change in accountability systems produced immediate and dramatic results. Simply by changing to the

new system the schools in New Mexico dramatically improved their standings. In the fall of the 2011-12

school year the last AYP Report was released by the state. By their account 811 schools or 98% of all the

public schools in the state had failed to make AYP. Just six months later, after being evaluated under the

school grading system, that number was just 64 failing, or F, schools; 8% of all the public schools in New

Mexico.

It is worthwhile to note the new accountability system makes an effort to hold schools accountable for what

they purport to do; help their students gain knowledge and skills. It also tries to protect schools from

elements that effect how well a student is learning but for which they are unable to control; such as

students speaking a language other than English or the negative effects of poverty. While the intricacies of

the calculations necessary to determine a school grade are well beyond lay people and most educators, its

sheer complexity suggests an effort to be fair. A characteristic very similar to NCLB but named and

calculated differently is the expectation that the students with the most severe barriers to learning improve

their educational standing. Under NCLB that is the students with disabilities, English learners, and students

hampered by poverty. In the school grading system it is the improvement of the bottom 25% of the student


population when ranked by academic achievement testing. In actuality it is the same students. The full

criteria and associated points possible are detailed in the following table.

Elementary and Mid School Grades Pts High School Grades Pts

Current Standing Performance in Math & Reading Percent Proficient. Students are tested on how well they met targets for their grade level.

40

Current Standing Performance in Math & Reading Percent Proficient. Students are tested on how well they met targets for their grade level.

30

School Growth In the past 3 years did schools increase grade level performance? For example did this year’s 3rd graders improve over last year’s 3rd graders?

10

Career and College Readiness Are students prepared for what lies after high school? Schools receive credit when students participate in college entrance exams, dual credit coursework, and coursework leading to vocational certification. Schools receive additional credit when students meet success goals.

15

Growth of Highest Performing Students The highest performing students are those whose scores place them in the top three quarters of their school. How well did the school help individual students improve? Individual student growth over the past 3 years is compared to average individual growth for the state.

20

School Growth of Highest Performing Students How does your school’s top three quartiles 11

th graders

performance improve over time? Each year it is a different group of 11

th graders.

15

Growth of Lowest Performing Students The lowest performing students are those whose scores place them in the bottom quarter of their school. How well did the school help individual students improve? Individual student growth over the past 3 years is compared to average individual growth for the state.

20

School Growth of Lowest Performing Students How does your school’s bottom quartile 11

th graders

performance improve over time? Each year it is a different group of 11

th graders.

15

Opportunity to Learn Does the school foster an environment that facilitates learning? Are teachers using recognized instructional methods, and do students want to come to school? This is a student opinion survey.

10

Opportunity to Learn Does the school foster an environment that facilitates learning? Are teachers using recognized instructional methods, and do students want to come to school? This is a student opinion survey.

8

Student and Parent Engagement Does the school provide exceptional encouragement for involving students and parents in education? Examples include community outreach and mentoring programs.

5

Student and Parent Engagement Does the school provide exceptional encouragement for involving students and parents in education? Examples include community outreach and mentoring programs

5

blank

Graduation How does the school contribute to on-time graduation? On-time means within 4 years, and within 5-years to a lesser extent. In 2012, 6-year success rates will also contribute.

17

Total 100 Total 100

By July 2012 PED revisited the school ratings for the 2011-12 school year. The reissued rating showed a

substantial decrease in the grades APS schools earned since the now “preliminary” grade release in January.

A B C D F

Elementary 0 18 25 31 15

Middle 0 12 5 10 0

High 2 5 7 5 0


These new grades were the catalyst to enact Instructional Audits, as required in the waiver from NCLB

requirements. The purpose of the Instruction Audits is to provide schools with a mechanism by which they

may examine their practices to determine where improvements might be required. In order for the audits

to be impartial and objective as possible teams of educators external to the school would visit the school for

several days to determine the evidence of a school’s practice. There were several required elements to

evidence determination:

1. Observation

2. Interviews with principal, leadership team, teachers, staff, students, parents and community

members

3. Additionally the school, through a committee process, was required to complete several

workbooks requiring extensive responses comparing the local school to the attributes identified

by PED as elements of a successful school

The specific documents to be completed and reviewed by the visiting team prior to the visit were:

1. School and System Reflective Summary

2. Literacy Reflective Summary

3. Math Reflective Summary

The expectation was that the schools would also respond to the audit findings that the audit teams would

also record in the Reflection documents and then articulate the improvements the school would make to

address the deficiencies found by the audit in the school’s on-line Educational Plan for Student Success

(EPSS). The improvements that were to be written had to align to the 7 Turnaround Principles, which PED

determined were critical necessities for the success of any Priority or Focus school.

1. Providing Strong Leadership: (1) reviewing the performance of the current principal (2) either replacing the principal if such a change is necessary to ensure strong and effective

leadership; or demonstrating to the SEA that the current principal has a track record in improving achievement and had the ability to lead the turnaround effort

(3) providing the principal with operational flexibility in the areas of scheduling, staff, curriculum, and budget

2. Ensuring that teachers are effective and able to improve instruction: (1) reviewing the quality of all staff and retaining only those who are determined to be effective

and have the ability to be successful in the turnaround effort (2) preventing ineffective teachers from transferring to these schools (3) providing job-embedded, ongoing professional development informed by the teacher

evaluation and support systems and tied to teacher and student needs 3. Redesigning the school day, week or year

to include additional time for student learning and teacher collaboration 4. Strengthening the school’s instructional program

based on student needs and ensuring that the instructional program is research-based, rigorous, and aligned with State academic content standards

5. Using data to inform instruction for continuous improvement by providing time for collaboration on the use of data


6. Establishing a school environment that improves school safety and discipline and addressing other non-academic factors that impact student achievement, such as students’ social, emotional, and health needs

7. Providing ongoing mechanisms for family and community engagement

The schools with an F designation, also called Priority Schools, were visited by contractors hired by PED. The

schools with D designations, called Focus Schools, were visited by district based personnel. Ultimately

district personnel visited 52 elementary and mid schools. No high schools required instructional audits. SIG

schools were waived from this process because of their strong relationship to PED. This number is larger

than the total number of D schools because district chartered charter schools became APS’s audit

responsibility and also the alternative schools. It should be noted that there are other mechanisms by which

a school could become a Priority School or Focus School that is not dependent upon their final letter grade.

During the month of September, 2012, sixteen teams of district teachers and central office staff visited the

52 schools, searching for evidence, conducting their interviews and reviewing the school responses in the

Reflective Summaries and adding their own findings as well.

Audit teams were expected to rank their concerns on a three point scale; 0 = of little concern, 1 = somewhat

concerning, and 2 = of significant concern and requiring immediate attention. Final audit reports were

prepared for each of the schools and submitted to district leadership, the principal of the school and PED.

Additional analysis by the RDA Data and School Systems Team, formerly known as the Accountability Team

(A-Team), was able to tease out common findings and frequently referenced areas of concern. The team

was able to group the findings thematically. The full matrix of common concerns is in the appendix.

However, most frequently cited significant concerns for schools are:

Elementary Schools

Instructional Program 1. Questioning/Higher Order Skills

2. Support for ELL

3. CCSS Implementation

Effective Instruction

4. Lack of or inconsistent application of differentiated instruction

Using Data

5. Data Analysis

6. PED assigned School Grade

7. Data Systems


Middle Schools

Instructional Program 1. Parent communication and engagement

2. Support for ELL/ESL students

3. Administrative presence, monitoring of classrooms and feedback

4. Use of inclusion/integrated model for Special Education support


5. Lack of differentiated instruction

6. Wider use of strategies to support multicultural/multilingual learners

7. Need for visible, challenging goals and standards for students

8. Absence of higher order questioning

Using Data

9. Cursory analysis of data to drive instruction

The relationship between the audit findings of significant concern and the 7 Turnaround Principles is

immediately apparent. The most referenced principles are Strong Leadership, Effective Teachers, and Using

Data. Also of concern are Strengthening the Instructional Program and Family and Community Engagement.

Also worth mentioning is the Common Core implementation in elementary schools. The notes associated

with this finding suggest that the implementation is struggling with little vertical articulation occurring,

internal coordination or change in pedagogy. Also, within the Using Data category is the PED assigned

School Grade. This rather broad statement may seem to have little bearing on the conduct of daily

instruction. Notes within the audits point to teachers that are unable to identify their Q1 students which are

those students most at risk academically and thus most needing teacher interventions and support, very

much a classroom based practice.

It would appear that district leadership has become aware of some of these issues through other means and

has been proactive in responding to the needs that have been identified. The curriculum and the

professional development departments invest a great deal in keeping principals informed regarding the

changing expectations for instruction and curriculum with the implementation of common core.

The professional development department works with the instructional coaches to provide them with a

level of expertise regarding the instructional shifts required pedagogically as Common Core comes into their

school. A new model for coaches is emerging that engages coaches and teachers to practice the new

instructional skills together. Additional reading coaches have been hired to provide additional targeted

support.

Language and Cultural Equity is piloting two new programs this year, one in reading complex texts and one

in math especially for the English learner. Both hold promise to provide the rigor and depth required in a

way that is supportive of the English learner.


The RDA Assessment Team is building new assessments that anticipate the rigor, research simulations, and

depth of knowledge required by the upcoming national assessments. The RDA Data and School Systems

Team has done some amazing work utilizing the AIMS system to deeply analyze student performance at the

individual, classroom, and school level, comparing DBA and SBA standards, benchmark and skill performance

to reveal strengths and weaknesses both in the learning and in the teaching. They continue to work in the

schools at either the leadership or instructional level utilizing the AIMS tools, the School Grade Workbook,

and the audit report to support school based improvements.

The school grade accountability system, while dramatically improving on the No Child Left Behind

accountability system continues to utilize testing to support accountability. In spite of test publisher and

psychometrician warnings regarding the use of assessments for purposes other than those intended, that

practice will continue. For schools that struggle to make an acceptable grade real learning is eschewed for

developing students that can perform well on reading and math tests. The new national standards will help

to address that.

New elements of accountability to be introduced in the near future will likely include automatic retention of

third grade students and student test scores contributing to teacher evaluations. APS will need to

proactively intervene where possible to protect the education of its students, the careers of its educators,

and the reputations of its schools as these new elements are developed and implemented by the state.

Appendix

1. Middle school key findings

2. Elementary school key findings

Middle School Instructional Audits – Key Findings

Project Name: D and F School Instructional Audits – Fall 2012

Level: Middle Schools (n = 10)

Summary: Differentiated instruction and improved parent involvement and communication were the two most mentioned issues identified by auditors. The absence of differentiated instruction was also the primary audit finding in elementary schools followed by only cursory use of data and data systems to drive instruction which is also a key finding in middle schools.

Instructional Program

These issues were identified with a ‘2’ indicating need for immediate attention

Number of mentions

Parent communication and engagement 7

More support for ELL/ESL students 3

Administrative presence, monitoring of classrooms and feedback 2

Use of inclusion/integrated model for SPED support 2

Direction from administration 1

More CCSS collaboration during PLC’s 1

PLC and Committee alignment 1

Low morale 1

Attendance and truancy issues 1



Number of mentions

Absence of differentiated instruction 7

Wider use of strategies to support multicultural/multilingual learners 3

Need for visible, challenging goals and standards for students 2

Absence of higher order questioning 2

Better classroom management 1

Consistent use of AVID strategies and Cornell notes 1

Improved lesson planning 1

Use of active learning and cooperative learning groups 1

Using Data


Number of mentions

Cursory analysis of data to drive instruction 5

Lack of understanding regarding PED school grade and components 1

Student understanding and tracking of own data 1

Elementary School Instructional Audits –

Key Findings

Project Name: D and F School Instructional Audits – Fall, 2012

Department: Elementary Schools (n=45)

Summary: The three most often mentioned areas of concern by auditors were: 1) an absence of differentiation in the classroom; 2) cursory use of data or data systems to inform instruction or support differentiation and 3) questioning strategies in classroom which do not engage higher order thinking skills.

These items were identified with a ‘2’ indicating need for immediate attention


Few Mentions

1-3

Commonly mentioned

3-7

Frequently Mentioned

8+ Issue

Questioning/Higher Order Skills

X

Questioning strategies which do not call for use of higher order thinking skills, active engagement and checking for understanding

Support for ELL X

Need for comprehensive system of support for ELL students

CCSS Implementation X

Uneven planning, implementation and integration of CCSS – missing vertical integration of curriculum across grade levels



Few Mentions

1-3

Commonly mentioned

3-7


8+ Issue

RTI/SAT X

RTI and SAT processes for identifying and supporting low performing students are ineffective, unclear or confusing

CCSS Communication

X

Lack of information about CCSS for students in classrooms and for parents

Home-School Communication

X

Better communication with parents regarding grade level expectations and markers of proficiency, ways to support homework, additional parent/family center ESL classes

Classroom & Time Management

X

Ineffective classroom or time management and need for school behavior management plan

Collaboration X No time for grade level collaboration

SPED X

Little recognition or engagement of Special Education students in classrooms

RTI Materials X

Limited or underutilized materials to support RTI literacy and math programs

Staff Communication X

Quicker and clearer communication about staff and district directives



Few Mentions

1-3

Commonly mentioned

3-7


8+ Issue

Collaboration X Ineffective use of collaboration time

SPED Programs X

Fidelity in using programs provided for Special Education students

Programs X

Too many programs and conflicts between CCSS implementation and us of ‘fidelity’ programs like America’s Choice – too many programs diverting attention from CCSS

Lesson Plans X

Lesson planning without a clear focus on standards or assessments of learning

Tier II Math X

Tier II math program inadequate for moving students to proficiency – programs take time away from core classroom instruction

Walkthroughs X

Inconsistent use of walkthroughs to provide specific feedback for improving instruction

Dual Language X

No clearly defined curriculum alignment plan for dual language program – DL not represented on Instructional Council

Instructional Coach

X Instructional Coach assigned duties which do not support classroom teachers



Few Mentions

1-3

Commonly mentioned

3-7


8+ Issue

Teacher Effectiveness X

Systems for measuring teacher effectiveness and setting high performance expectations are not in place/Systems for acculturating and supporting new teachers is needed

Technology X Uneven or limited skills using technology in the classroom

Systems X

Need for consistency in goals, expectations, beliefs, practices, and expected outcomes across the entire school

Items identified with a ‘2’ indicating need for

immediate attention


Few Mentions

1-3

Commonly mentioned

3-7


8+

Lack of or inconsistent application of differentiated instruction

X The most widely noted issue across all elementary audits

Future Goals X Students unable to make connections between learning and future goals i.e. high school/college

Attitudes X Attitudes towards BP and ED programs as interfering with educational needs of all students

Items identified with a ‘2’ indicating need for

immediate attention

Using Data

Few Mentions

1-3

Commonly mentioned

3-7


8+

Data Analysis

X

Limited analysis and use of data to drive/differentiate instruction – including baseline data from spring assessments used at the beginning of the year

PED Grade X Limited knowledge of factors contributing to PED-issued school grade – including Q1 membership

Data Systems

X

AIMS and Data Director systems are used seldom or infrequently to access student information and drive instruction. Program or curriculum-based data information systems are used more commonly

ACCESS X ACCESS data is not being reviewed or used to help guide instruction

Data Protocols

X Protocols for using data to set goals, monitor program effectiveness and student learning are not being created

Commentary: Results from the audit reports complicate our assumptions about the role data play in our district. The phrase ‘data-driven culture’ is widely acknowledged as a fundamental component to turn-around strategies and higher performing schools yet there is scant evidence that data, particularly data from DBA and SBA exams, is being used in any substantive way to differentiate instruction and address specific learner-centered problems. APS may want to consider engaging teachers in a dialogue about these questions and audit findings, to step back from commonly held assumptions about the differentiation, higher order questioning strategies in the classroom and the uses of data and information systems put in place meant to support the work of teachers in APS.

Documents

ALBUQUERQUE PUBLIC SCHOOLS - aps.edu · easily comprehended by the lay public. ... students speaking a language other than English or the negative effects of poverty. While the intricacies