Embed Size (px)
Citation preview
INTER-RATER AGREEMENT IN KANSASSummer Principals Academy
July 22-24, 2014
Abilene, KS
2
What is Inter-Rater Agreement?• Consistent
understanding• Consistent
agreement• Consistent
process• Consistent
credibility
The degree 2 raters using the same scale give the same rating in identical situations.
3
Where is Inter-rater Agreement Useful??
Inter-Rater Agreement is useful in all areas of administrative practice including:
• Kansas Educator Evaluations• Kansas Accreditation • Personnel Decisions• Family Engagement Programs• Budget Management
4
What are the Goals of Inter-rater Agreement?
• The Inter-Rater Agreement Statistical Goal is 75% - agreement with no rating more than one level apart.
• The Goal is to have an administrative Common Frame of Reference for your district such as:• Agreement with no rating more than one level apart. • Consistent understanding• Consistent agreement• Consistent process• Consistent credibility
5
How is Inter-rater Agreement Helpful??
• What is a Common Understanding?
• How does Inter-rater Agreement help administrators??
• Facilitates Quality• Provide Consistent Messaging• Impacts Increased Student Growth• Enhances Trust • Facilitates Quality Feedback• Legally Defensible • Assists Accurate Data Collection• Allows for Increase Cooperation
over Time • Passes Public Scrutiny
6
District-wide Agreement about External Criteria
• Agreed Upon External Criteria for:• Observations• Artifacts• Units of Study• Processes• Program Development• Time Limits • Walk-Throughs• Protocols
7
Pragmatics/ConsiderationsPotential Landmines
• Discuss with an Elbow Partner Possible Issues that may cause Rater Error• Leniency: Benefit of the
Doubt• Central Tendency: Safety in
the Middle• Comparing One with
Another: Stick to the Rubric• Relying on Gut too Much:
Stick to the Rubric• Evaluating Absent
Evidence: Stick to What is Observed
8
How to Train Administrators in using Inter-rater Agreement
• Inter-Rater Agreement Training – Things to Consider: • Synchronous and Asynchronous • Experience of the Participants• How to Ensure Fidelity • Use of Video Examples• Use of Case Studies• Bias Exists• Single versus Multiple Observations• Professional Judgment• Degree of Familiarity
9
Getting Started in YOUR District
• Getting Your Bearings• Developing a Guidebook• Developing a Training Protocol
10
I. Getting Your BearingsPilot with Administrative Team1. Pilot Approach: Start with Safe Example:
• Use a video from YouTube• Assess Specific Area of Teaching
• Student Engagement• Teacher Activity• Classroom Environment
2. Review current issue/goal/vision/needs
3. Review current practices/instrumentation• Rubrics• Protocol• Training
11
II. Developing a Guidebook• Paper and Electronic: Synchronous and Asynchronous• EXCELLENT Administrative Council Endeavor –
Ultimately to be used during Professional Learning Times with the Staffs
• Pieces and Parts• Terms/Definitions: Frame of Reference/Common Understanding
• “Demonstrates”• “Frequently”• “Usually”
12
II. Developing a Guidebook• Pieces and Parts (continued)
• Examples and Rubrics• Areas to Observe
• Establishes/maintains a culture of safety, respect and rapport
• Knows how to make content accessible to all learners
• Plans standards-based instruction and formative assessments
• Uses a variety of instructional strategies to meet needs
• Deepens understanding through critical thinking, problem-solving and student reflection
• Analyzes student performance and provides feedback
• Collaborates to support student learning
13
II. Developing a Guidebook• Pieces and Parts (continued)
• Exemplars: Video and Artifacts• Master-Coded Videos: Leads to high level of
agreement• Example Artifacts
• Classroom Rules• Lesson Plans• Parent Note Home
• Lesson Plan for Adults: Andragogy
14
III. Developing a Training Protocol• Step-by-Step Procedure
1. Initial Training: Goals/Terms• Practice Interpreting Rubrics• Differentiate between Different Performance Levels with Examples
2. Calibration of the Players• Video and Artifacts
3. Review of the System
4. Adjustments (If needed)
5. Implementation
6. Evaluate/Reflect/Adjust
7. Re-Calibrate…
15
• Start Slowly• Start with
“Engagement” with a Small Group
• Get Agreement• Gain Confidence
Over Time• Write and Refine in
Teams
Take Aways – To Do List
16
• Start with the End in Mind• Develop Common
Understandings• Use Groups to Ensure
Buy-In• Build Measurable Steps to
Reach Each Goal
Final Thoughts
