School Improvement Grant Reviewer Training - misn-cdn.resa…misn-cdn.resa.net/uploads/videos/3099/SIG_III_Peer_Reviewer... · 5/27/2014 2 June 4, 2014 SIG Peer Reviewer Training

5/27/2014

1

Office of Education Improvement and InnovationSchool Improvement Support Unit

School Improvement Grant Peer Reviewer Training

5/27/2014

2

June 4, 2014

SIG Peer Reviewer Training

2

The purpose of this webinar is to:

▫ Orient prospective SIG application reviewers before they engage in the peer review process

▫ Refresh experienced SIG application reviewers on the process

▫ Provide potential and current SIG applicants with an understanding of the application review process and the timelines associated with each step

Webinar Purpose

5/27/2014

3

June 4, 2014


3

Participants will be able to:

▫ Describe the peer review process for the School Improvement Grant (SIG) competition

▫ Understand how to review and score SIG applications▫ Distinguish between the roles of the School Improvement Support (SIS) Unit staff & Peer Reviewers

Expected Outcomes

5/27/2014

4

June 4, 2014


4

Federally‐funded School Improvement Grant review requires:

▫ A fair process

▫ Transparency

▫ Separation of authority between the administration of the federal grant and the application review process (ergo peer review)

▫ Leveraging the expertise of the public school stakeholders

Purpose of Peer Reviews

5/27/2014

5

June 4, 2014


5

Application Window•MDE Solicits Peer Reviewers

Application Deadline•MDE finalizes Peer Review Teams

Within 5 Days after Application Deadline•View webinar•Applications and rubric sent to peer reviewers

2 Week Review Period•Peer Reviewers independently read and score assigned applications

Three Weeks after Application Deadline•Consensus Review Meeting

Process Overview

5/27/2014

6

June 4, 2014


6

• Peer reviewers are vetted through resume review• Peer reviewers are VOLUNTEERS• Reimbursement for mileage and overnight stay (if reviewer must travel over 50 miles one way) • Reviewers are assigned to teams based on experience, employment status, and subject matter expertise• Peer reviewers must sign a form stating that they have no conflict of interest with the specific applications they review• Peer reviewers will be eligible to receive up to 7 SCECHs.

SIG Application Peer Reviewers’ Responsibilities

5/27/2014

7

June 4, 2014


7

▫ MDE

▫ School and district Administrators

▫ Public School Academy community

▫ Public school board members

▫ ISD Personnel

▫ Teachers

▫ Parents

▫ School consultants

▫ External Partner Providers

▫ Current and past SIG subgrantees

SIG Application Peer ReviewersPeer Reviewers include representatives of different stakeholder groups, such as:

5/27/2014

8

June 4, 2014


8

• In most cases, reviewers will score 4 or 5 applications, including a common application that will be used as part of a norming exercise

• Applications are assigned randomly although the SIS Unit will attempt to avoid potential conflicts of interest

• Reviewers will need to sign a statement regarding the absence of conflict with the applications they review

SIG Application Review—Process

5/27/2014

9

June 4, 2014


9

• SIS Unit will form teams to include representatives from different stakeholder groups

• After individually reviewing SIG applications, 3 or 4 person peer review teams will meet to reach consensus on scores

• Further details of the Consensus Review Meeting will be sent along with applications.

Peer Review Teams

5/27/2014

10

June 4, 2014

10


• Bring copies of all applications assigned and individually scored with rubrics and notes to the face‐to‐face consensus review meeting

• Bring conflict of interest disclosure form• Participate in the full‐day, face‐to‐face consensus review

meeting• Work as a team with other peer reviewers to reach

consensus on each application• Reach a consensus score on all questions for all

applications assigned• Contact SIS Unit if there is a conflict of interest with the

applications that you have been assigned

Consensus Review Meeting

• Volunteer by notifying MDE SIS Unit• Provide a resume to MDE SIS Unit• Review this webinar• Review, sign, and submit to the SIS Unit the conflict of

interest disclosure • Read and score each assigned application individually,

include comments on the scoring rubric• Notify MDE SIS Unit if you are assigned an application with

potential conflicts of interest

Before Consensus Review Meeting

Peer Reviewer Role

5/27/2014

11

June 4, 2014


11

• Facilitated and moderated by SIS Unit staff members

• Exercise starts with the first section of the rubric related to the “common” application, reaching a consensus rubric score

• Assign the remainder of rubric sections for the “common” application to one of the review teams

Consensus Norming Exercise

5/27/2014

12

June 4, 2014


12

• Each team is provided a consensus scoring rubric for each application to be scored

• The state employee on each team will record the team’s results on an electronic rubric for each application

• Consensus scores will be saved to a flash drive and submitted to the SIS Unit before the end of the meeting

Peer Review Teams

5/27/2014

13

June 4, 2014


13

• Facilitate/moderate the process• Answer technical questions• Collect flash drives with electronic rubric scores & comments• Verify & rank the scores• Ensure every score lower than 10 has a comment regarding how the answer could be improved• Collect completed conflict of interest disclosure forms• Distribute and collect evaluation forms

SIS Unit Role During Consensus Review

5/27/2014

14

June 4, 2014


14

• Applications with the highest scores will be recommended for a subgrant as the funds allow

• No announcement is made until all of the recommendation letters are signed by the State Superintendent and returned to the SIS Unit

Recommending Awards

5/27/2014

15

June 4, 2014


15

The review rubric includes elements related to:▫ The analysis of needs (school and district)▫ Existing district capacity▫ Alignment with approved State Reform/Redesign plan▫ Budget, timeline, and goal setting▫ Evidence of commitment▫ Plans for administering and evaluating the impact of SIG implementation▫ Sustainability plans

Reviewing SIG Applications—Rubric

5/27/2014

16

June 4, 2014


16

• 36 different plan elements organized in 8 sections

• Awarding less than full points for any element must be explained

• Elements of the rubric can earn varied points depending on the question

• Reviewers must choose one of the defined point categories

Reviewing SIG Applications

5/27/2014

17

June 4, 2014


17

• Review rubric before reading applications

• Read and re‐read applications before scoring

• Score each application individually prior to the Peer Review Consensus Meeting

• Take NOTES! Whether on a hard copy or electronically, jot down your thoughts to remind yourself WHY you scored the application as you did

• Come prepared to explain/defend your scores

Reviewing SIG Applications

5/27/2014

18

June 4, 2014


18

• Did the narrative response/graph address the application requirement• Did the narrative/response stay within the page limit• Was the scoring rubric used to formulate the response to the application requirements• Was the response clear, concise, and succinct• Was the response specific and detailed when necessary• Did the response align to the application requirement• Were the responses confusing, incomplete or lacking clarity, logic, or common sense

10 Points– Exemplary Answers

5/27/2014

19

June 4, 2014


19

• Section A1: LEA Application, Requirement 1 – Analysis of Need

Describe the process the LEA used to analyze the needs of the school and how the intervention was selected for each school.

LEA Application Example

5/27/2014

20

June 4, 2014


20

Staff identified in the Priority School, including building leadership teams, principals, school improvement team members and parents, met several times with the Director of School Improvement to conduct a review of data relevant to this school improvement effort. The leadership teams and school improvement teams consist of teacher representatives from all grade levels and content areas in each school along with parent volunteers and building administrators. Parents actively participated on each team as full and engaged members in the decision making process. The following data were reviewed: Comprehensive Needs Assessment (CNA), MEAP, Scantron Performance Series assessments, grades, student attendance, student behavior, and parent/staff surveys, in order to analyze the needs of the school.

Based on the data from the past three years, it is evident that the school has made progress toward meeting or exceeding targets toward their goals and assuring that all students leave prepared for success at the high school level, much work still needs to be done. In recent years, the school has been meeting achievement targets in Reading and Mathematics. However there is a need to improve attendance in order to meet the achievement targets overall. In addition, the school continues to lag behind both district and state proficiency levels in nearly all assessment categories. Studies of longitudinal data from state assessments and nationally norm‐referenced assessments indicate, with some notable exceptions, that while students show growth as they move through the schools year‐by‐year, their growth generally slows.

Each of the four intervention models allowed under MCL 380.1280(c) were discussed by staff from the building, district administrators, the Education Association (union), parent groups, open public meetings at the school, and among school board members during their regularly scheduled meetings. Based on all of these discussions and a review of the data, it was agreed that the Transformation Model would best meet the needs of the school to bring about the desired school reform; improved teacher effectiveness and increased student achievement. This model will allow staff to develop necessary skills through improved leadership practices, embedded professional development aligned to the specific needs of staff and students in each school, and a focus on results.


5/27/2014

21

June 4, 2014


21

LEA Application ExampleSection A1: LEA

Application Exemplary Implemented Partially Implemented Getting Started Score:Total Possible: 140 10 7 3 1 0

1. Analysis of Need The plan details a comprehensive assessment system that uses longitudinal and annual dataaligned with clearly defined student performance targets. The system was designed with stakeholder input and is evaluated annually. The plan provides disaggregated data timely and accurate information that is useful to improve student performance targets. There is evidence that multiple sources of data were used to inform strategies to close the achievement gap, all stakeholders were an integral part of the process and a detailed description around the selection of the intervention model is provided, including how building staff and other stakeholders were involved in the selection.

There is a comprehensive assessment system that uses longitudinal and annual data is reviewed periodically and aligned with defined performance targets. The plan provides disaggregated data timely and accurate information that is useful to improve student performance targets. There is a description of the process used to select the reform model and how teachers and leaders were involved.

There is a comprehensive assessment system alignedwith defined performance targets. They are in theprocess of developing a framework for disaggregating data to close the achievement gap. There is a description of the process and dataused to select the reform model.

They are beginning to develop an assessment system aligned to performance targets. Disaggregated data is provided to the schools and it is unclear how they selected the reform model.

5/27/2014

22

June 4, 2014


22

Staff identified in the Priority School, including building leadership teams, principals, school improvement team members and parents, met several times with the Director of School Improvement to conduct a review of data relevant to this school improvement effort. The leadership teams and school improvement teams consist of teacher representatives from all grade levels and content areas in each school along with parent volunteers and building administrators. Parents actively participated on each team as full and engaged members in the decision making process. The following data were reviewed: Comprehensive Needs Assessment (CNA), MEAP, Scantron Performance Series assessments, grades, student attendance, student behavior, and parent/staff surveys, in order to analyze the needs of the school.

Based on the data from the past three years, it is evident that the school has made progress toward meeting or exceeding targets toward their goals and assuring that all students leave prepared for success at the high school level, much work still needs to be done. In recent years, the school has been meeting achievement targets in Reading and Mathematics. However there is a need to improve attendance in order to meet the achievement targets overall. In addition, the school continues to lag behind both district and state proficiency levels in nearly all assessment categories. Studies of longitudinal data from state assessments and nationally norm‐referenced assessments indicate, with some notable exceptions, that while students show growth as they move through the schools year‐by‐year, their growth generally slows.

Each of the four intervention models allowed under MCL 380.1280(c) were discussed by staff from the building, district administrators, the Education Association (union), parent groups, open public meetings at the school, and among school board members during their regularly scheduled meetings. Based on all of these discussions and a review of the data, it was agreed that the Transformation Model would best meet the needs of the school to bring about the desired school reform; improved teacher effectiveness and increased student achievement. This model will allow staff to develop necessary skills through improved leadership practices, embedded professional development aligned to the specific needs of staff and students in each school, and a focus on results.


5/27/2014

23

June 4, 2014


23

A. What was done well in the application:1. The application identified members of the team that reviewed the data around school improvement. 2. The application identified a variety of data that was reviewed to meet the schools needs.3. The application indicated that the staff reviewed the intervention models.

B. What would have strengthened the application:1. If there was information included in the application that identified how the assessments mentioned would be

aligned to and adjusted to help improve and meet the student performance targets.2. If there was data provided in the application along with how the disaggregated data will be available to the

building and continually supported by the district. 3. If the applicant had provided a timeline or calendar identifying when data would be made available to schools.4. If there was data that supported the selection of the intervention model; such as was a vote taken, the

number of staff that participated, any agendas or the percentage of time spent on analyzing the models.

It appears that the target focus of improved attendance was not addressed.Please note that there was a great deal of information provided in the narrative, but it was too vague and did not directly answer the question.

Comments and Feedback

5/27/2014

24

June 4, 2014


24

• Section A2: School Application, Requirement 11 – Sustaining Reform

Describe how the reforms from the selected intervention will be sustained in this school after the funding period ends.

School Building Application Example

5/27/2014

25

June 4, 2014


25

The district is committed to the success of this transformation plan which focuses on the development of human capital, distributed leadership, research‐based practices in instruction and assessment and financial and operational flexibility. By designing a data‐driven assessment and instructional improvement process, and by developing the skills of the building leadership and staff in implementing these processes, we have built a redesign plan that will have long‐lasting and sustainable impacts beyond the grant cycle. The Office of School Improvement, building principals, building leadership teams and external consultants will collaboratively develop plans for sustaining and monitoring the effectiveness of the redesign project beyond the SIG timeline. This plan will provide metrics on the effectiveness of the redesign efforts both within the school and on achievement gains made by students after they leave the school. In addition, both the evaluation and sustainability plans will be used to inform redesign efforts that may need to be undertaken by other schools in the district, in future years. The development of a distributed, building‐level decision making process along with the training of building level leadership teams, school improvement teams and Professional Learning Communities provides for a systemic, sustainable approach to improved professional practices and improved achievement for all students. Beyond structural changes in the clock schedules, proposals for extended learning activities under this plan are designed to strengthen school‐community partnerships and develop a school culture that respects learning. Once established, our goal is to be able to maintain programmatic changes with General Fund and/or Section 31a funds. Therefore, it is expected the district will be able to sustain the reforms as stated in this plan, after the grant cycle, through reallocation of existing funds and by focusing on the development of its human capital through effective professional development within a distributed leadership model.

Was the requirement addressed?


5/27/2014

26

June 4, 2014


26


11. Sustaining Reform The plan detailed and identified how the building will increase capacity and they have created a sustainability plan. There is evidence that the district has a commitment to retaining effective building leaders and staff beyond the grant cycle.

The plan describes capacity building and how the plan will be sustained overtime.

The plan describes capacity building, but does not include long term sustainability.

Plan does not address capacity or sustainability.

Comment:

Section A2: School Exemplary Implemented Partially Implemented Getting Started Score

Total Possible: 150 10 7 3 1 0

5/27/2014

27

June 4, 2014


27

The district is committed to the success of this transformation plan which focuses on the development of human capital, distributed leadership, research‐based practices in instruction and assessment and financial and operational flexibility. By designing a data‐driven assessment and instructional improvement process, and by developing the skills of the building leadership and staff in implementing these processes, we have built a redesign plan that will have long‐lasting and sustainable impacts beyond the grant cycle. The Office of School Improvement, building principals, building leadership teams and external consultants will collaboratively develop plans for sustaining and monitoring the effectiveness of the redesign project beyond the SIG timeline. This plan will provide metrics on the effectiveness of the redesign efforts both within the school and on achievement gains made by students after they leave the school. In addition, both the evaluation and sustainability plans will be used to inform redesign efforts that may need to be undertaken by other schools in the district, in future years. The development of a distributed, building‐level decision making process along with the training of building level leadership teams, school improvement teams and Professional Learning Communities provides for a systemic, sustainable approach to improved professional practices and improved achievement for all students. Beyond structural changes in the clock schedules, proposals for extended learning activities under this plan are designed to strengthen school‐community partnerships and develop a school culture that respects learning. Once established, our goal is to be able to maintain programmatic changes with General Fund and/or Section 31a funds. Therefore, it is expected the district will be able to sustain the reforms as stated in this plan, after the grant cycle, through reallocation of existing funds and by focusing on the development of its human capital through effective professional development within a distributed leadership model.

Was the requirement addressed?


5/27/2014

28

June 4, 2014


28

A. What was done well in the application:1. The application identified how they are moving toward sustainability beyond the grant cycle. This included a data‐driven

assessment and instructional improvement process and developing the skills of the leadership and team.2. The application identified the use of metrics to determine the effectiveness of the redesign efforts on the school and the

achievement gains of the students. 3. The application indicated that if needed this is a replicable model for the district. 4. The application indicated a distributed leadership model with training for the team.5. The application identifies extended learning opportunities.6. The sustainability will occur through programmatic changes and reallocation of general funds and 31a funds.

B. What would have strengthened the application:1. The application would have earned a 10 on the rubric if there was information included on the tools, techniques and

strategies used to build the capacity of the leaders.2. The application needed to included information regarding the tools used to support the leaders and the creation of a

talent pool and the length of time it will take to build it.3. The application needed to include information on the district professional development plan for leadership

Comments and Feedback

5/27/2014

29

June 4, 2014


29

• Jill Thompson, Secretary• Kristine Davidson, Consultant• Bill Witt, Supervisor

[email protected]

MDE SIG Website

School Improvement Support UnitSIG III Grant Points of Contact

Documents

School Improvement Grant Reviewer Training - misn-cdn.resa…misn-cdn.resa.net/uploads/videos/3099/SIG_III_Peer_Reviewer... · 5/27/2014 2 June 4, 2014 SIG Peer Reviewer Training