Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Using Cost-Effective Processes to Develop Large-Scale Data-Driven Continuous Improvement Systems for Local Programs
Laurie A. Van Egeren, Jamie Wu, Michigan State University Angelina Garner, Charles Smith, David P. Weikert Center for Youth Program Quality
American Evaluation Association Minneapolis, MN October 26, 2012
21st Century Community Learning Centers
Michigan 21st CCLC
In 2011-2012:
TACSS is Quality Assurance for: • 320 elementary, middle and
high schools sites • 40,000+ students • Over $50M investment
Evaluation Scope in 2003
Al I modern domestic dogs are descend ants of the Gray Wolf
Evaluation Scope Now
Continuous Improvement
Assess
Plan Improve
Hmprowment
Quality Improvement Support System
MDE State Education Agency
MSU State Evaluator
TACSS Program Quality
Improvement
Standard Indicators of Quality
Cost-Effective Data Reports
Cost-Effective Data Reports
Capacity-Building for Data Use
1. Standard quality indicators
Identify Indicators MDE
Literature TACSS
Advisory MSUBoard
I.II 1. Instructional Context
2. Organizational Context
3. Positive Relationships
Select Data Sources
Youth survey Parent survey Staff survey
Supervisor survey
Administrator report
Observational program self-assessment
Attendance/ activity data
(web)
School outcomes data
Comparability: Weighted 10- pt scale
Measure Weight Academic activity participation 1.5 Homework help/tutoring participation for academically at-risk students
1.5 Academic enrichment participation 1.5 Activities informed by grade-level content standards 1 Student reports of academic support quality 1.5 Academics is top priority .5 Supervisor connection to school-day content 1 Staff connection to school-day content 1.5 Total 10
10
Instructional Context State Grantee
5.1 6.5 4.7 4.6 5.94.4 6.9 4.3 4.1 5.20
2
4
6
8
Enrollment/ Academic Enrichment Connections Instructional Retention Content Content to School Quality
0
2
4
6
8
10 10-pt scale for comparability
II State
Grantee – State Comparisons
State Grantee
Grantee – State Comparisons
Site variations at a glance
Go deeper – comparisons on measures State Org Sites
Go deeper – comparisons on measures
0-10 pt indicator score
Measure scores however defined
Go deeper – comparisons on measures Sites
Uh oh…
Even deeper – item data for sites
1-gradle stud1ents at thi.s s ite and statewide w·ho agre 1ed or ng.a.gement in the program.
ent of Students who Agrered or S
Your si'te Statewide 46°/o 77°/o 50°/o 23°/o s1°10
62°/o 77°/o
Yo1ur site co . a .. to
state Very· low
Very· low
Very low
Very· low
!·nts_ Statewide numbers are for students in the san1e grade' se1Ved b ·
Uh oh…
MI 21•1 Century Commu nity Lellrning Ce nters Leading Indicator s Report
Citi' P ublic Schoolc Peril 7, 2010
Overa• Ft lcture Compared to :State
• -O rtp11iut ional Cont ext
si.t~itv lt!roimt~ Gtart.. St• SUl'f eo,,w,, .. ,>11, rttentOI' ll'III\IQf!n'el'tAH.IWII e:>Pffl'"(e/ ~·ene,i
•="1
'
Instructional Context
Acade mic Instructio n 5 .6 5 .3 5.2
Cc nnection to schoo lb 7l.°A. 80% Yes Formal poUcy t'or connecting with l '1ach u s
129( "" No
FUl -time Sit e ( oord inato? 631< 100% Yes Ac~d"mic ~ct ivity
811< 70% 67% participatiOnc
Provi~ion of homework hel p( 43% 63% 67%
Provis ion ot a cademic 53% 47% 5 1%
~nrich~ntc
Provis ion of t utorin, ~ 11% 0% 0% A.t:i::1Je111k.")b lup pli1..11ilyd 861< 77% 60%
Ce rtif ieo teache rs provide 4:J':-E °" ~
ct1,;dJ l:'111i1,;,up pur l<
St 11 riP nt rf'pnn,; n f ;tr:MPmic 53% 59% 50%
support quality'
5 .8 , .. ... 5 .1
Yes Yes Yes No
No Ne No No
Yes Yes Yes Yes
89" 82% LS% 761<
89% 71% 36% 60%
"" 78% 33% 68% -Q)(, 0% 0% 0)(; n. ...................... _ ... _.,..r .......................................... .. -....---........ --~-~ .... ......
701< 50% 100% 67%
m, "" ~ "" 81% 50% 27% 88%
\l- ...... - - ..,.~-n.. .............. ,._...,,~.-1:-................................ . =..----....---.......... --..---.-_ ....... ,_.... .... ,...,_. ......... _ _,, __
inoo•--•-• .-..:'.='--·.:::::~------::. ... ..;::.-.--
o..---.c:..,,c..~ ... o.--n...w. ................. .,, ...... t" ................. _ .... ____ ......... ==-~...._ ................ _,---.,., .. ____
'"'*" ..,.,._"'___... ... 0.- ...._...___..._
- ~-NM~ ·-·--· -~--==:-s:... ... .:=..--.. ~-.. :'-..--., .. -----..--~----··-.....-
Grantee Summary
Site Comparisons
Site Details
2. Cost-effective local report production
Assumption: You’re analyzing
data anyway
Process 1. Collect data 2. Develop report template in Word
3. Analyze data to match [decisions] 4. Create excel or .csv file of data 5. Use Word mail merge to populate
reports (tweak if necessary) 6. Voila!
Step 1: Collect Data
Youth survey Parent survey Staff survey
Supervisor survey
Administrator report
Observational program self-assessment
Attendance/ activity data
(web)
School outcomes data
Step 1: Collect Data
Youth survey Parent survey Staff survey
Supervisor survey
Administrator report
Observational program self-assessment
Attendance/ activity data
(web)
School outcomes data
MI 21•1 Century Commu nity Lellrning Ce nters Leading Indicator s Report
Citi' P ublic Schoolc Peril 7, 2010
Overa• Ft lcture Compared to :State
• -O rtp11iut ional Cont ext
si.t~itv lt!roimt~ Gtart.. St• SUl'f eo,,w,, .. ,>11, rttentOI' ll'III\IQf!n'el'tAH.IWII e:>Pffl'"(e/ ~·ene,i
•="1
'
Instructional Context
Acade mic Instructio n 5 .6 5 .3 5.2
Cc nnection to schoo lb 7l.°A. 80% Yes Formal poUcy t'or connecting with l '1ach u s
129( "" No
FUl -time Sit e ( oord inato? 631< 100% Yes Ac~d"mic ~ct ivity
811< 70% 67% participatiOnc
Provi~ion of homework hel p( 43% 63% 67%
Provis ion ot a cademic 53% 47% 5 1%
~nrich~ntc
Provis ion of t utorin, ~ 11% 0% 0% A.t:i::1Je111k.")b lup pli1..11ilyd 861< 77% 60%
Ce rtif ieo teache rs provide 4:J':-E °" ~
ct1,;dJ l:'111i1,;,up pur l<
St 11 riP nt rf'pnn,; n f ;tr:MPmic 53% 59% 50%
support quality'
5 .8 , .. ... 5 .1
Yes Yes Yes No
No Ne No No
Yes Yes Yes Yes
89" 82% LS% 761<
89% 71% 36% 60%
"" 78% 33% 68% -Q)(, 0% 0% 0)(; n. ...................... _ ... _.,..r .......................................... .. -....---........ --~-~ .... ......
701< 50% 100% 67%
m, "" ~ "" 81% 50% 27% 88%
\l- ...... - - ..,.~-n.. .............. ,._...,,~.-1:-................................ . =..----....---.......... --..---.-_ ....... ,_.... .... ,...,_. ......... _ _,, __
inoo•--•-• .-..:'.='--·.:::::~------::. ... ..;::.-.--
o..---.c:..,,c..~ ... o.--n...w. ................. .,, ...... t" ................. _ .... ____ ......... ==-~...._ ................ _,---.,., .. ____
'"'*" ..,.,._"'___... ... 0.- ...._...___..._
- ~-NM~ ·-·--· -~--==:-s:... ... .:=..--.. ~-.. :'-..--., .. -----..--~----··-.....-
Grantee Step 2: Develop report Summary template
Site Comparisons
Site Details
Step 3: Analyze for report – Decisions! Indicator MI Org 1 2 3
1.4 Connection to School Day 4.7 4.1 4.4 6.9 4.2
Formal policy for connecting with school daya,b
69% 75% 75% 75% 75%
Supervisor communication with schoole 46% 11% 0% 0%
Staff communication with schoold 27% 21% 40% 25%
School investment in programb 61% 80% Yes Yes Yes
Step 3: Analyze for report – Decisions! Indicator MI Org 1 2 3
1.4 Connection to School Day 4.7 4.1 4.4 6.9 4.2
Formal policy for connecting with school daya,b
69% 75% 75% 75% 75%
Supervisor communication with schoole
46% 11% 0% I
in
mpute to get dicator
0%
Staff communication with schoold 27% 21% 40% score 25%
School investment in programb 61% 80% Yes Yes Yes
Step 3. Analyze for report – Decisions!
What is minimum N
(varies)
How determine cut-offs?
Use syntax!
Step 4. Create Excel or .csv file
Step 5. Mail merge Excel file into template
Step 5. Mail merge Excel file into template
3. Capacity-building for data use
MICHIGAN~
EdDeparonenc1t~~ ._.,, ) uca ton
A Decade in the Making
Why TACCS?
TACSS Goals
1. Grow a culture of performance accountability
2. Develop a low-stakes infrastructure for continuous quality improvement
3. Improve overall quality of 21st CCLC services and start up for new sites
4. Improve the instructional quality for young people
Important Concepts Underpinning TACSS P
OLI
CY
SE
TTIN
G
OR
GA
NIZ
ATIO
N S
ETT
ING
INS
TRU
CTI
ON
AL
SE
TTIN
G
Instructional Quality
Management Skills for Continuous Quality Improvement
Low Stakes Accountability
TA/Coach Values & Methods
Important Concepts Underpinning TACSS Management Skills for CIP
PLAN
IMPROVE ASSESS
• Lead a team to • Carry out plan to assess the quality improveof instruction • Lead team to create instructional quality
• Provide real-time an improvement • Monitor progressstaff performance plan based on data and repeat feedback • Select align
methods training for direct staff
Important Concepts Underpinning TACSS High Stakes Accountability Policy
Objective Data Publicity Action Improved
Outcomes
Important Concepts Underpinning TACSS
Objective Data
MeaningfulInformation
Action/ Expertise
Improved Outcomes
Low Stakes Accountabilities
LearningCommunity
ImprovementEfforts
Low Stakes Accountability Policy for CQI
TACSS Project Model in Detail 5-year project
5.5 FTEs (1 manager, 4 TA/Coaches, 1 support staff)
1 PTE (Contract Coach) Regional
TA Coaches
Improve Service Quality
& Child Outcomes
The TACSS Model Comprehensive Support Sequence
MDE Kickoff Event Introductory
meeting with Grantee
Data Profile assembled
Onsite visit, data profile review, and
prep for TAPlanning Day
Director Interview Regarding CQI Practices & red
flag issues
Team Self Assessment of
Instructional Quality
Planning with Data sessions; Develop
TA-Plan
Maintenance of TA Plan with on-going TA/coach support
Data Driving the System Leading Indicators to Program Improvement
Grantee Profile
Site Profiles
Site Detail
Technical Assistance Plan
• Co-created • Linear/sequential • Accountability • Intentionality • Scheduling • Use of Data to drive decision
making • Living/Working document
Core & Supplemental Services Menu
The TACSS Model
POLICY CONTEXT
ORGANIZATIONAL CONTEXT
INSTRUCTIONAL SETTING
Hig
her I
nten
sity
.
.
.
.
.
Comprehensive Supports are Multi-level
. School district and union issues around staffing
Understanding vendor and partnership relationships
Training for Conflict Resolution
Support continuation grant /renewal
Support program self-assessment
Site visits to provide quality coaching
Lower Intensity
TACSS Calendar Year July Aug Sept Oct Nov Dec Jan Feb March April May June July Aug
Letters Coach's reflection letters Personal invitation to kick off orientation TACSS orientation at Kick off Introductory TACSS meeting
TA Planning Self Assessment process support ( YPQA, PIP) External Assessment scheduling/observation Leading Indicator Introduction/review (PD) External Assessment review Data planning session (support PD to lead staff) Mission is Possible professional development opportunity Monthly follow up communications
To Sum Up •Leading Indicators = roadmap to quality program
•Founded in mass-reported data •Decisions about changes are driven by data
•Technical assistance supports programs to use that data in ways they identify
Questions…