Upload
uriel-mann
View
24
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Measuring Performance within School Climate Transformation Grants. Rob Horner OSEP TA-Center on PBIS www.pbis.org. Goal. Define resources for measuring performance within SCTG grants that may improve efficiency, consistency and effectiveness of evaluation efforts. - PowerPoint PPT Presentation
Citation preview
Measuring Performance within
School Climate Transformation Grants
Rob HornerOSEP TA-Center on PBIS
www.pbis.org
Goal• Define resources for measuring performance within
SCTG grants that may improve efficiency, consistency and effectiveness of evaluation efforts.
• Provide a brief summary of schools using PBIS in the U.S.
Why a Framework for Multi-tiered Behavior Support?• The fundamental purpose of a Multi-tiered
Behavior Framework is to make schools more effective and equitable learning environments.
Predictable
Consistent
Positive
Safe
All the parts
Logic Model
Intervention Strategies/Programs
Implementation Process
Fidelity of Implementation
Student/ Family Outcomes
Multi-tiered
Why the strategies
work
Build Capacity to Scale
and Sustain
Are we doing the
strategy/program
Valued outcomes/ Academic, Behavior,
Social Emotional
Four ways the OSEP TA-Center may be
helpful
• Coordination and Communicationo Webinars and information dissemination
• Intervention Contento Tier Io Tier IIo Tier III
• Measuring Fidelity of Multi-tiered Behavior Support
• Measuring Student Outcomes
Aligning multiple
initiatives begins by defining common
fidelity and outcome measures
Resources
• Grantees are held accountable for what they submitted. • But some consistency across projects may help all evaluation efforts
• Resources that may be helpful:
o Evaluation Blueprint (www.pbis.org)o Evaluation Annual Reports (www.pbis.org)o State and District Capacity Assessments
• www.sisep.orgo Multi-tiered Fidelity Assessment (Tier I, Tier II, Tier III)
• www.pbisapps.orgo Multi-tiered student discipline data systems
• SWIS, CICO-SWIS, ISIS-SWIS• www.swis.org, www.pbisapps.org
Performance Indicators: SEA
• The number of training and technical assistance events provided by the SEA School Climate Transformation Grant Program to assist LEAs in implementing a multi-tiered behavioral framework.
• The number and percentage of LEAs provided training or technical assistance by the SEA School Climate Transformation Grant Program that report an improvement in knowledge and understanding of the implementation of a multi-tiered behavioral framework.
• The number and percentage of schools in LEAs provided training or technical assistance by the SEA School Climate Transformation Grant Program that implement a multi-tiered behavioral framework.
Performance Indicators: LEA• Number and percentage of schools that report an annual decrease
in office disciplinary referrals.
• Number and percentage of schools that report an annual improvement in the attendance rate.
• Number and percentage of schools that report an annual decrease in suspensions and expulsions, including those related to possession or use of drugs or alcohol.
• Number and percentage of schools annually that are implementing the multi-tiered behavioral framework with fidelity.
Performance Measure
Considerations
• Use a Formal Evaluation Model
• Measure Fidelity at All Three Tiers
• Consider Measuring Improved State/ District Capacity
• Measure student outcomes within a formal “decision system”
Plan
Perform
Measure
CompareSchool-wide
Positive Behavior Support
Context
InputFidelity
Impact
Multi-Tiered
Behavior Support
Evaluation Model: Evaluation Blueprint/
www.pbis.org
SchoolsAre Schools Implementing Multi-tiered Systems of Behavior
Support?
Possible Measures of Multi-Tiered Behavior Support Fidelity
Measures (www.pbis.org)
Tiered Fidelity Inventory (TFI) Tier III
Individual Student System Evaluation Tool (ISSET)
Benchmark of Advanced Tiers (BAT)
Measure of Advanced Tiers Tool ( MATT)
Tiered Fidelity Inventory (TFI) Tier II
Tiered Fidelity Inventory (TFI) Tier I
School-wide Evaluation Tool (SET)
Team Implementation Checklist (TIC)
Benchmarks of Quality (BoQ)
Regardless of the
Intervention approach used,
TA-Center Fidelity measures
are available and
appropriate
Available from OSEP TA-Center www.pbis.org
No Cost
Assessors Training PowerPoint available
Assessors Training Video in development
TFI- Technical Adequacy(McIntosh, Massar et al. 2014)
Content Validity (.95; .93; .91)Usability (12 of 14 > 80%) (15 min per Tier)Inter-rater Agreement (.95; .96; .89)Test-retest reliability (.98; .99; .99)Factor Analysis (in process)
TFI Data Tier I: 2014
DR GL M RB H0
10
20
30
40
50
60
70
80
90
100
Jan-14 May-14 Oct-14
DR GL M RB H0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
67%71%
83%
62%54%
73%
88%
77%
46%50%
Tier II Score on TFI
Winter 2013 Spring 2014
DR GL M RB H0%
20%
40%
60%
80%
100%
56%
79%
88%
71%
32%
79%
91%
71%65%
44%
Tier III Score on TFI
Winter 2013 Spring 2014
Tier II
Tier III
Action Planning
State/Region Capacity
Does a state have the capacity to implement multi-tiered systems of behavior support?
Is Technical Assistance improving the capacity of a state/region to implement multi-tiered systems of behavior support?
Possible Measures of SEA/LEA Capacity
www.sisep.org
• State Capacity Assessment (SCA)o Assessment of extent to which state department has capacity to
implement evidence-based practices at scales of social significance?o No Cost, from OSEP’s SISEP TA-Center www.sisep.org;
www.scalingup.orgo Scores: Total, Sub-scale, Itemo Used for initial assessment, action planning, progress monitoring
• District Capacity Assessment (DCA)o Assessment of extent to which a school district (or region) has capacity
to implement evidence-based practices as scales of social significance?o No Cost, from OSEP’s SISEP TA-Centero Scores: Total, Sub-scale, Itemo Used for initial assessment, action planning, progress monitoring
Implementers Training Video
SCA/ DCA Format(Team with Coach/ Assessor)
Item Data Sources Scoring Criteria Score 0 1 2
MinnesotaState
MinnesotaState
MinnesotaState
State SCA Item Analysis: Action Planning
District Capacity
Is Technical Assistance improving the capacity of a district to implement multi-tiered systems of support?
District Capacity
District Capacity Assessment
District Capacity Assessment
District Capacity Assessment: Duda, et al. 2012
District Capacity Assessment
District Capacity Assessment: Duda, et al. 2012
District Capacity Assessment
District Capacity Assessment
Measures of Student Benefit
• Office Discipline Referralso Tier I = SWISo Tier II = CICO-SWISo Tier III = ISIS-SWIS
• School Climateo Under development
• School Safety Surveyo Risk Factor/ Protective Factor
Performance Monitoring Summary
• Consider including measures of MTBSF fidelity at all three tiers.o Many Tier I optionso Several Tier II optionso One new measure that does all three (Tiered Fidelity Inventory (TFI)
• Consider assessing SEA/ LEA “Capacity to implement evidence-based interventions”o State Capacity Assessmento District Capacity Assessment
• Outcome measureso Office discipline referrals (SWIS, CICO-SWIS, ISIS-SWIS)… latero School Climateo School Safety Survey
Excellent Examples of
State and District
Reports at www.pbis.or
g