M3-Review-SLOs-13NOV13

The Student Learning Objective (SLO) Process is comprised of three (3) components: Design, Build, and Review. Student Learning Objectives provide a valid assessment of teacher effectiveness through student performance outcomes based on standards.

Welcome to Training Module 3: “Reviewing”

Concept

Within the Review Module, the trainer will guide participants to implement a multifaceted, three-level review of Student Learning Objectives, including identified performance measures.

Key Points for Trainers

1. Explain in macro-terms the “big picture” of how SLOs are but one component of a much larger teacher effectiveness system.a. As required by Act 82, an Educator Effectiveness Rating Tool must contain measures based on teacher observation and practice and

multiple measures of student performance. Student Performance data will include the following: Building Level Data (15%), Teacher Specific Data (15%), and Elective Data (20% for teachers with eligible PVAAS data, 35% for teachers without eligible PVAAS data).

b. This Educator Effectiveness Rating Tool functions as a framework for the evaluation and summative process for classroom teachers.2. Clarify that the Review Module will guide participants through a three-level quality assurance review of the created Student Learning Objective

and the identified Performance Measures. The Review phase helps ensure that the operational and technical expectations of the process are met or exceeded.

3. Ensure participants understand that this phase is the quality assurance component, focusing on issues that range from completing the SLO Process Template correctly to examining the validity evidence of a vendor-made assessment/performance measure.

Learning Activity

___________________________________________________________________________________________________________________________

Technical Notes“Structure”

1. Concept – “What is this slide telling the audience?”

2. Key Points – “What/Where are the details ‘needed for teaching’?”

3. Learning Activity – “How can the participant’s learning be enhanced?” (This item will not be populated for every slide.)

Module 3-SLO Reviewing 1

Student Learning Objective

Concept

As you work through the Review Process, you will need to access the following materials:1) Handout #4 – Quality Assurance Checklist-SLO (found in SLO/Review/Guides)2) Handout #5 – SLO Procedural “Cheat Sheet” (found in SLO/Review/Guides)3) Template #6 – SLO Coherency Rubric (found in SLO/Review/Templates)4) Performance Measure Rubric for Teachers (found in ALS/Review/Stuff)5) Refinement Control Checklist (on-demand Performance Measures only, found in SLO/Review/Stuff)


1. Assist the participants in conceptualizing that the SLO Process Template is organized into two major components: SLO Goals and Performance Indicators on page 1 of the template, and Performance Measures and Expectations on page 2.

2. Clarify that the comprehensive evaluation of a new performance measure cannot be fully completed until after the measure is administered and data is collected. The current tools focus on the development and administration of the measures, with additional training being developed for the “post-administration” review. For vendor-made measures, there should be sufficient evidence on the inferences of scores, reliability, etc. that also includes how the performance measure was developed.

3. Stress to participants that the Review process includes a quality assurance check on each Performance Measure selected to ensure that they are aligned with the content standards, free of bias and sensitive content, and developmentally appropriate.

Learning Activity

1. Have participants go to www.pdesas.org to retrieve the five items mentioned above from the SAS Portal /Homeroom learning portal. Also make sure they have the completed SLO Process Template (Template #4) found in SLO/Build/Templates.

2Module 3-SLO Reviewing

Process Components

Concept

This slide reviews the parts associated with each phase of the SLO Process. This Module will focus on the Review Phase of the SLOs.


1. It will be important for each educator to have all the completed templates from their work with Module 1: Design and Module 2: Build. That would include the following templates:

a. Template #1: Goal Statement (found in SLO/Design/Templates)b. Template #2: Targeted Standards (found in SLO/Design/Templates)c. Template #3: SLO Blueprint (found in SLO/Design/Templates)d. Template #4: SLO Process Template (found in SLO/Build/Templates)e. Template #5: Performance Task Framework (found in SLO/Build/Templates)

2. A key point for teachers is that the Review phase is iterative, meaning that quality assurance criteria are designed to inform corrective action rather than simply assign a score. Likewise, the performance measure quality requires at least one full administration to determine basic psychometric characteristics of newly developed assessments. Validity evidence builds upon successful iterations of the assessment “lifecycle.”

Learning Activity


Session Objectives

Concept

The three-tier quality assurance review process of each SLO includes the following:• Completeness

• Using the “Quality Assurance Checklist” (Handout #4) and the SLO Procedural “Cheat Sheet” (Handout #5), educators will determine the completeness of the SLO Process Template (Template #4).

• Comprehensiveness• Using the Performance Measure Rubric (found in ALS/Review/Stuff) and the SLO Procedural

“Cheat Sheet” (Handout #5), educators will evaluate the quality of each teacher-made performance measure.

• Coherency• Using the SLO Coherency Rubric (found in SLO/Review/Templates) and the SLO Procedural

“Cheat Sheet” (Handout #5), educators will evaluate the entire SLO Process.


1. Emphasize that all performance measures are evaluated in terms of high quality and rigor. The Performance Measure Rubric for Teachers (found in ALS/Review/Stuff) will guide educators through each phase of the build-out; Design, Build, and Review.

2. Reinforce the 3 C’s concepts (Completeness, Comprehensiveness, Coherency) so the participants understand the reason each aspect (“C”) must be considered before statements regarding “high quality” SLO can be made and supported.

Learning Activity

1. Have the participants spend 5-10 minutes reviewing and discussing in their small groups the different tools and how they are used in evaluating an SLO. After this exploration, have the members select a model and “test drive” Handout #4 (Quality Assurance Checklist-SLO).


Task Structure

Concept

The first step in the Review Process is to check for the completeness of the SLO Process Template. This step ensures that the SLO Process Template is complete. While reviewing, any incomplete, incorrect, or missing element will be highlighted and potential corrective actions will be identified. The “Help Desk” (found in SLO/Build/Stuff) provides detailed information on filling in the template.

The second step in the Review Process is to check the comprehensiveness of the performance measures selected and/or developed. Educators will be guided through a process to review each aspect of the Performance Measure: Design, Build, Review (checking for quality). The Performance Measure Rubric-Teacher (found in ALS/Review Stuff) and Quality Assurance Screening Tool are helpful. For vendor-made assessments, a more detailed rubric is available.

The final step in the Review Process is to check the coherency of the SLO Process. Coherency is about examining the “parts” of a teacher’s SLO and verifying the alignment to the Big Idea. The SLO Coherency Rubric (found in SLO/Review/Templates) focuses on examining how the components fit into a cogent process for applying evidence of student achievement to teacher evaluation.


1. Clarify that when evaluating the entire SLO Process (coherency) educators will be checking to ensure that everything is aligned with the Goal Statement and Targeted Content Standards. Handout #5 SLO Procedural “Cheat Sheet” and the SLO Coherency Rubric will aid educators through the process and identify any issues that will need to be addressed.

2. Emphasis that completing the form correctly is a relatively straight forward task and the evaluation of a performance measure’s technical data is much more complex and time consuming. It (Comprehensiveness) should be viewed as a two stage process for both the SLO and the performance measures within the SLO: before administration and after administration.

Learning Activity


Design Coherency

Concept

This figure is designed to help visualize how the “parts” fit into the “whole.”


1. Explain that this figure is designed to help visualize how the “parts” fit into the “whole.” It is essential for the audience to fully understand how only a coherent SLO approach should be used within the greater Educator Effectiveness System.

• Goal Statements are the “big idea” upon which the SLO is based. These goals must be based upon Pennsylvania standards and provide a rationale for choosing a particular standard which addresses the important student learning.

• Performance Indicators are the expected levels of achievement for students in the SLO population.

• Performance Measures are the various tools/assessments which will be used to measure student achievement toward a specific goal.

2. A quality discussion can emerge from this slide by having the participants identify validity threat associated with student samples, assessment quality, and indicator rigor. It is essential to the “Elective” rating’s validity characteristics that systematic errors are not introduced into the SLO process at either the component level or in how the components are aligned to the SLO Goal Statement.

Learning Activity


Helpful Tools

Concept

As noted previously, many resources are available for educators to use during the Review process. They include the following.1. Handout #4 – Quality Assurance Checklist-SLO (found in SLO/Review/Guides) 2. Handout #5 – SLO Procedural “Cheat Sheet” (found in SLO/Review/Guides)3. Template #6 – SLO Coherency Rubric (found in SLO/Review/Templates)4. Performance Measure Rubric for Teachers (found in ALS/Review/Stuff)5. Refinement Control Checklist (on-demand Performance Measures only; found in SLO/Review/Stuff)All training documents and materials can be found on the SAS Portal and RIA’s Homeroom learning portal. Go to http://www.pdesas.org and you will see a link to RIA’s Homeroom portal as well.


1. Differentiate how tools can assist in the building task:a. Guides have materials such as handouts, rules of thumb, model SLOs, etc. that reinforce

content presented in the videos. In this module, models are provided to guide new users in what a completed SLO, including developed and/or selected performance measures.

b. Templates are used to complete each phase of the process. The key template is SLO Process Template 10.0 (available as a Word document-Template #4- and as an online tool in SLO/Build/Templates); however, Template #5 is provided (found in SLO/Build/Templates) for performance tasks that may be culminating events over multiple days (e.g., student projects). Other development templates are found within the Assessment Literacy Series/Review.

Learning Activity

1. Have participants retrieve and review the Handouts and Templates associated with the ReviewProcess.


Final Deliverables

Upon completion of Step 3: Review; educators will have a completed SLO Process Template that identifies the rigorous, high-quality performance measures as well as any area that was identified through the process needing improvement.


1. Articulate the expected deliverables for teacher-made performance measures:a. Purpose Statementb. Targeted Standardsc. Specifications and Blueprintsd. Scoring Keys and Rubricse. Operational Forms (tests/tasks)f. Administrative Guidelines

2. Expected deliverable for vendor-made performance measures:a. This deliverable will depend on the measure adopted by the school and/or district (i.e., Dibels, AIMS

Math, etc.); however, the vendor will often have written evidence and data supporting the measurement claims. Educators should simply reference this information, thus demonstrating that the selection of the measure meets the intended need of the educator (i.e., to measure the targeted content standards within the “Big Idea”).

3. Using the Performance Measure Rubric for Teachers (found in ALS/Review/Stuff), the QA Screening Tool (found in ALS/Review/Stuff) and the Quality Assurance Checklist (Handout #10, found in ALS/Review/Guides), participants will be able to identify/document any shortcoming found and determine corrective actions needed.

Learning Activity

1. Probe the audience for their understanding of what is meant by a “rigorous performance measure” using the below set of questions:

a. To what degree does the performance measure align with targeted standards?b. What is the cognitive demand of the tasks/items, both individually and in totality? (Refer to Depth of

Knowledge Chart found in ALS/Build/Guides and DoK Chart II found in ALS/Build/Stuff).c. Is the performance measure’s reading level and vocabulary age and grade appropriate?d. Are the targeted skills appropriate?e. Are the items/tasks clear, concise, and understandable?f. Are the items/tasks free of bias?g. Are the items/tasks sensitive to different cultures, religions, ethnic and socio-economic groups?


Generic Review Workflow

Concept

This slide provides a general overview of the workflow for the Review Process.

“Work flow” within this context means the step-by-step Review process needed to ensure high-quality SLOs have been created.


1. Step 1: Packet Inventory –a. Checking to make sure that all required documents are included in the packet for review.

i. Template #4 – SLO Process Template (found in SLO/Build/Templates)ii. Template #5 – Performance Task Framework (for each Performance Indicator; found in SLO/Build/Templates)iii. Teacher-developed Performance Measures, specifications/blueprints, scoring sheets, administrative guides

2. Step 2: Review for Completeness – Using the SLO Procedural “Cheat Sheet” (Handout #5; found in SLO/Review/Guides) and the Quality Assurance Checklist-SLO (Handout #4; found in SLO/Review/Guides), participants will:

a. Review each section of the SLO Process Template to make sure it has been completed correctly.b. Highlight any missing or incomplete elements.c. Circle any incorrect element and outline the discrepancy.d. Recommend corrective actions.

3. Step 3: Evaluate SLO Template for Comprehensiveness – Using the Performance Measure Rubric for Teachers (found in ALS/Review/Stuff), participants will evaluate the quality of each teacher-made performance measure. This rubric is comprised of 18 technical aspects of quality assessment organized into three (3) strands; Strand 1: Design, Strand 2: Build, Strand 3: Review. Participants will rate each element and list supporting technical evidence, provide further explanations and details where needed, and make any recommended actions for shortcomings.

a. Strand 1: Design – has five (5) elementsb. Strand 2: Build – has six (6) elementsc. Strand 3: Review – has seven (7) elements

4. Step 4: Evaluate Performance Measure Quality – Another step in assuring performance measure quality is to use the following:a. QA Screening Tool (found in ALS/Review/Stuff)

a. Part I – Screeningi. Ensures that all deliverables are included for the performance measure.

b. Part II – Form/Item Rigori. Ensures that the measure is developmentally appropriate.ii. Ensures that items are aligned with targeted content standards.iii. Ensures items have been screened for sensitivity, bias, and fairness.

c. Part III – Qualityi. Ensures that specifications/blueprints are aligned with operational forms.ii. Administrative guidelines are clear and standardized.iii. Accommodation guidelines are provided/referenced.iv. Scoring guidelines are standardized.v. Scoring rubrics provide detailed scoring information.

b. Quality Assurance Checklist (Handout #10; found in ALS/Review/Guides))i. Provides a review for specifications and blueprints, scoring key and rubrics, operational forms, and administrative guides to ensure that performance measures meet

the criteria for high-quality assessments. 5. Step 5: Determine Rating/Score Coherency – Using the SLO Coherency Rubric (found in SLO/Review/Templates), participants will review the entire SLO Process. This rubric has several technical

requirements divided into three (3) Strands: Design, Technical, and Quality. Through this review process, participants will check for alignment among all the elements (goal statement, targeted content standards, performance indicators, performance measures), identify any issues, and make recommendations for corrective actions.

a. Strand 1: Design – has five (5) elementsb. Strand 2: Technical – has six (6) elementsc. Strand 3: Quality – has four (4) elements

6. Step 6: Identify Corrective Actions – There are several places where corrective actions can be identified and documented throughout this process. The following documents can be used for this purpose:a. SLO Procedural “Cheat Sheet” (Handout #5)b. SLO Coherency Rubric (Template #6)c. Quality Assurance Checklist-SLO (Handout #4)d. Performance Measure Rubric for Teacherse. Quality Assurance Checklist (Handout #10)f. QA Screening Tool

Learning Activity


Modified (with calibration) Review Workflow

Concept

The Train-the-Trainer approach allows Pennsylvania to “calibrate” additional trainers to ensure consistency throughout the state. It provides an opportunity for district professional development staff, school administrators, and lead teachers to thoroughly understand the SLO Process and ensure that it is being implemented in classrooms across the state with fidelity.


1. Once educators have completed the Review procedure and if they have identified any corrective actions, it is imperative that a process on how and when those identified corrective actions will occur is determined.

2. Dimension 1: Template for Completeness a. Apply a quality assurance checklist to the developed materials to ensure all documents meet the requirements.

3. Dimension 2: Template for Comprehensivenessa. Take the Performance Measure Rubric for Teachers (found in ALS/Review/Stuff) and evaluate the quality of each

teacher-made performance.b. Reviewing forms involves a content review; a review for bias, sensitivity, fairness, and alignment. c. Content review:

• Determine if each item/task clearly aligns to the targeted content standards.• Evaluate all items for accuracy.• Evaluate if items are age-appropriate.• Ensure that each item and answer options are clear and understandable.

d. Item Bias:• Bias-free items/tasks provide an equal opportunity for all students to demonstrate their knowledge and

skills.e. Fairness and Sensitivity:

• Ensure items are sensitive to different cultures, religions, ethnic and socio-economic groups, and disabilities.

• Ensure there is a balance of gender roles.• Ensure that positive language, situations, and images are used for performance measures.• Ensure that items avoid text that may elicit strong emotions in specific groups of students and prevent

those students from accurately demonstrating their skills and knowledge.4. Dimension 3: Rating/Score Coherency

a. Using the SLO Coherency Rubric, the entire SLO Process is evaluated to determine whether the technical requirements have been met.

5. Dimension 4: Performance Measure Qualitya. Applying a quality assurance checklist to the developed performance measures to ensure all documents meet the

basic principles of quality assessment.

Learning Activity


Calibration Procedures

Concept

The Calibration Procedures increase standardization among SLOs while strengthening SLO review teams’ technical skills. Participants should have the skills and knowledge related to the basic principles of quality assessment practices.

Quality Assurance is a critical step in the SLO Process. It is important that procedures are in place to ensure that SLOs and Performance Measures are selected/developed properly. Quality Assurance procedures provide a way for reviewers to do that.


1. Participants will review each packet through four (4) different lenses (dimensions). It is important to identify any missing or incomplete areas and develop recommendations for corrective actions. If needed, seek clarifications from facilitators. Misinterpretations of the SLO criteria or any element throughout the process can lead to unintended consequences (i.e., developmentally inappropriate performance measures being used with students is both unfair to the students and to the teachers who will be using the student outcome data to determine their Elective rating on the Teacher Effectiveness Rating Tool).

Learning Activity


Summary

Concept

During Module 3 participants reviewed, refined, and corrected the SLOs and the Performance Measures associated with each Performance Indicator. At this point, teachers should have all completed templates that were required throughout the entire SLO Process.


1. This professional development provided participants with a better understanding ofseveral quality assurance practices used in the SLO Process.

2. As noted previously, the SLO process was created to meet the need of Act 82, which requires PA to have an Educator Effectiveness Rating Tool that contains measures based on teacher observation and practice as well as multiple measures of student performance. SLO is the process that develops an Elective Rating through evidence of student achievement.

3. The SLO Process also assists educators in developing/selecting fair, valid, high-quality measures to evaluate student achievement.

Learning Activity


Business

M3-Review-SLOs-13NOV13