Upload
joanna-fowler
View
218
Download
1
Tags:
Embed Size (px)
Citation preview
Denise Huang
CRESST/UCLA
November 23rd to 25th, 2010
Giessen, Germany
International ConferenceNetwork on Extracurricular and Out-of-School Time Educational Research
Out of School Time in America
2 / 27
• Safe Haven
• Build Resiliency
• Opportunities for Enrichment
• Closing the Gap
• Academic Achievement
The Evolutions of Afterschool Programming for At-Risk Youths
3 / 27
THEORETICAL FRAMEWORK
4 / 27
A New Day For Learning
“A comprehensive , seamless approach to learning that values the distinct experiences that families, schools, after-school programs, and communities provide for children.”
A Report from the Time, Learning, and Afterschool Task Force 2007
5 / 27
The Ecological Model
Afterschool programs are social organizations embedded with cultural values coming from families, school, and
neighborhood climate and norms.
Community/Neighborhood
School Afterschool Program
Family
Child
6 / 27
Provisional Features of Positive Developmental Settings (Larson et. al)
Physical and Psychological Safety
Clear and consistent structure
Supportive relationships
Opportunities to belong
Positive social norms
Support for efficacy and mattering
Opportunities for skill building
Integration of family, school, and community efforts
7 / 27
Core Positive Youth Development Constructs
Work Place
Family
Congregation
Programs
Peers
Neighborhood
School Developmental Strengths
View of the Child
Promotion of Health Well-
being Thriving
Reduction in High-Risk Behaviors
Context
Person
Developmental Success
Community
Search Institute Insights & Evidence, November 2006
8 / 27
Leveraging Social Capitals in Afterschool Programs
Relationships
Expectations &
Social Norms
Trust
Staff & Student Relationships
Student Engagement
Organizational Capital
Social Capital
Intellectual Capital
Internal Networks
External Networks
Innovation and Staff
Development
Competence
Attitudes
Problem-solving
& Agility
9 / 27
Theoretical Logic Model for the Afterschool Partnership Study
Program SupportManagement
Staff /ResourcesPeriodical Evaluation
for Continuous Improvement
Periodic Assessment
of Student
Performance
Linkage to School Day
Linkage to School Day Research
Based Practice & Strategies
Research Based
Practice & Strategies
Content Practice Quality
After School
Program Quality
Content Aligned
to Standards
Content Aligned
to Standards
Set Goals
Align activitiesto goals
MotivationEngagement
Opportunities to Practice
(Attendance)
Content
Structure Process
10 / 27
The CDE Indicator Model
Instructional Features
ProgramEnvironment
Program Orientation
Staff Efficac
y
Setting Features
Expectation
Aspirations
Positive Youth
Development
Academic
RelationshipsSafety
Goals
Management
External Connectio
ns
Evaluation System
Alignment
Program Climate
Satisfaction
STAR
CASHEE
School Attendan
ce
Monitoring
Monitoring
Resources
Professional
Development
Student Engageme
nt
11 / 27
EVALUATION
12 / 27
The Need of Evidences to Support Expansions
Do afterschool programs contribute to positive academic development?
Do afterschool programs contribute to positive youth development?
What aspects of the program functioning contribute to these positive outcomes?
13 / 27
Prevalence of Different Evaluation Research
Process Evaluations:
Formative studies
• Evaluations that assesses the conduct of the program during the initial design and testing stages with the intent to improve the program
Program monitoring
• Systematic examination of program coverage and delivery- (target population, fidelity, efficiency)
• Identifying successful implementation strategies for program diffusion
14 / 27
Outcome Evaluation
Summative Evaluation
• Summative evaluation provides information on the product's efficacy ( it's ability to do what it was designed to do)
• By looking at the intervention group, the evaluator can examine the learning materials and learning process together with the outcomes-- hence the name Summative Evaluation.
Impact Evaluation
• Impact evaluation involves constructing a counterfactual
• Random selection and isolation from interventions are seldom practicable and sometimes ethically difficult to defend.
• Quasi-experimental method is often used.
15 / 27
Indicators for Program Effectiveness
Student attendance (in regular school and afterschool programs)
Performance measures (achievement tests, homework completion, classroom grades, language re-designation, school retention, future aspirations, etc.)
Non-cognitive measures (safety, attitudes towards school, relationships with adults, social competence, conflict resolution skills, self-esteem, and self-efficacy etc.)
Parent involvement
Professional development
Long-term effects (drop out rate, life satisfaction, etc)
16 / 27
Challenges in After School Studies
Diversity of program characteristics
Self-selectiveness
-Consent forms
Comparison groups
-Ethical issues
Transience
Availability of archived data sources
Meaningful outcome measures
Importance of dosage
17 / 27
COGNITION & ACHIEVEMENT
18 / 27
Common Outcome Measures for Cognition and Achievement
Academic achievement scores
Attitudes towards school/learning
Development of study skills (time management, organization, memory, etc. )
Development of academic enabler (self-efficacy, motivation, effort, etc.)
School dropout
Future aspirations
19 / 27
High Scope Afterschool Quality and Day School Outcomes 2007
Safe Environment
Safe Environment
Student Engagement
Autonomy &Opportunities to
reflect and practice
Student Engagement
Autonomy &Opportunities to
reflect and practice
Reading Achievement
Reading Achievement
***Program
Attendance
***Program
Attendance
Positive Interaction
Positive Interaction
School Suspensions
School Suspensions
20 / 27
Relationship between Program Attendance and Youth Outcomes
-0.100
-0.080
-0.060
-0.040
-0.020
0.000
0.020
0.040
0.060
0.080
0.100
2002-03 2003-04 2004-05 2005-06
Es
tim
ate
d M
ath
A
ch
ieve
me
nt G
row
th F
rom
Ba
se
lin
e
Time
Over 100 days
51 to 100 days
21 to 50 days
0 to 20 days
LA's BEST Attendance
21 / 27
ASES Program Attendance
0%
5%
10%
15%
20%
25%
30%
1-9
10-
19
20-
29
30-
39
40-
49
50-
59
60-
69
70-
79
80-
89
90-
99
10
0-10
9
11
0-11
9
12
0-12
9
13
0-13
9
14
0-14
9
15
0-15
9
16
0-16
9
17
0-17
9
18
0-18
9
19
0-20
0
Pe
rce
nt
of
Stu
de
nts
Days Att ended
After School Program AttendanceElementary Attendance (N=281,054) Middle School Attendance (N=146,292)
22 / 27
ASSETs Program Attendance
0%
10%
20%
30%
40%
50%
60%
70%
1-9
10-1
9
20-2
9
30-3
9
40-4
9
50-5
9
60-6
9
70-7
9
80-8
9
90-9
9
10
0-10
9
11
0-11
9
12
0-12
9
13
0-13
9
14
0-14
9
15
0-15
9
16
0-16
9
17
0-17
9
18
0-18
9
19
0-20
0
Pe
rce
nt
of
Stu
de
nts
Days Attended
After School Program AttendanceHigh School Attendance (N=73,902)
23 / 27
Improved School Attendance and Engagement in Learning
Elementary school students attending LA’s BEST afterschool program improved their regular school day attendance and reported higher aspirations regarding finishing school and going to college. Additionally, LA’s BEST participants are 20 percent less likely to drop out of school compared to matched nonparticipants. (UCLA National Center for Research on Evaluation, Standards and Student Testing, June 2000, December 2005 and September 2007)
A New Hampshire statewide study of students participating in academically focused afterschool programs, including those funded by the federal 21st Century Community Learning Centers Program (21st CCLC), found that more than half of regular attendees improved both behaviorally and academically. (RMC Research, 2005)
Ninety-two percent of high exposure of Citizen Schools participants were promoted on time to the tenth grade compared to 81 percent of matched nonparticipants. This is critical, since earning promotion to tenth grade on time is a key predictor of high school graduation (i.e. preventing drop out). (Policy Studies Associates, Inc., December 2006)
24 / 27
Improved School Attendance and Engagement in Learning-High School
Ninth grade students who formerly participated in The After-School Corporation (TASC) in middle schools had higher daily school attendance and credit accumulation than matched nonparticipants. (Policy Studies Associates, Inc., October 2007)
Participants in the Breakthrough Collaborative program enroll in college-preparatory mathematics courses at double the national average, and are accepted to college-preparatory high schools by more than 80 percent. (Breakthrough Collaborative, 2006)
High school students participating in Chicago's After School Matters program—which offers paid internships in the arts, technology, sports, and communications to teenagers in some of the city's most underserved schools—have higher class attendance, lower course failures and higher graduation rates than similar students who do not participate in the program. (University of Chicago, Chapin Hall Center for Children, 2007)
25 / 27
Improved Test Scores and Grades
Annual performance report data from 21st CCLC grantees across the country demonstrate that students attending 21st CCLC programs improve their reading (43%) and math grades (42%). (Learning Point Associates, November 2007)
The Promising Afterschool Programs Study, a study of about 3,000 low-income, ethnically-diverse elementary and middle school students, found that those who regularly attended high-quality programs over two years demonstrated gains of up to 20 percentiles and 12 percentiles in standardized math test scores respectively, compared to their peers who were routinely unsupervised during the afterschool hours. (Policy Studies Associates, Inc., 2007)
Participants in North Carolina’s Young Scholars Program with at least 280 hours in the program averaged double-digit increases annually for proficiency in both math and reading. Promotion rates rose by 38 percent. Furthermore, the number of Young Scholars receiving A’s and B’s increased an average of 38 percent, while the number receiving F’s decreased an average of 50 percent. (Z Smith Reynolds Foundation, 2006)
Active participants in programs offered by The After-School Corporation (TASC) were more likely to take and pass the Regents Math Sequential 1 exam by ninth grade than were nonparticipants. Thirty-two percent of active ninth grade participants took and passed the exam, compared to one percent of ninth grade nonparticipants. Fifty-two percent of active participants took and passed the Math Sequential 2 and 3 exams, compared to 15 percent of nonparticipants in the same grades. (Policy Studies Associates, Inc., 2004)
Participants of St. Paul Minnesota’s 21st CCLC Pathways to Progress program received better grades in English and math than nonparticipants. (University of Minnesota, Center for Applied Research and Educational Improvement, March 2004)
26 / 27
Keeping Kids Safe and On Track for SuccessA meta-analysis of 73 afterschool evaluations concluded that quality programs were consistently successful in producing multiple benefits for youth including improvements in children's personal, social and academic skills, as well as their self-esteem. (University of Illinois at Chicago, Collaborative for Academic, Social, and Emotional Learning, 2007)
Children attending LA’s BEST Afterschool program are 30 percent less likely to participate in criminal activities than their peers who do not attend the program. Researchers estimate that every dollar invested in the LA’s BEST program saves the city $2.50 in crime-related costs. (UCLA National Center for Research on Evaluation, Standards and Student Testing, September 2007)
Youth attending 23-40 or more days of Maryland’s After School Opportunity Fund Program showed a more positive gain on commitment to education and academic performance, and a reduction in delinquency and contact with the police. (University of Maryland, Department of Criminology and Criminal Justice, June 2004)
Teens who do nothree times more likely to skip classes , use marijuana or other drugs, drink alcohol, smoke cigarettes and engage in sexual activity.t participate in afterschool programs are nearly (YMCA of the USA, March 2001)
27 / 27
Helping Families
Parents who are concerned about their children’s after-school care miss an average of eight days of work per year. Decreased worker productivity related to parental concerns about after-school care costs businesses up to $300 billion per year. (Brandeis University, Community, Families and Work Program, 2004 and Catalyst & Brandeis University, December 2006)
In an evaluation of LA’s BEST, three quarters of the parents surveyed indicated that since enrolling their children in the program, they worried significantly less about their children’s safety and had more energy in the evening. (UCLA Center for the Study of Evaluation, June 2000 and December 2005)
Parents in the TASC study said that the program helped them balance work and family life: 94 percent said the program was convenient, 60 percent said they missed less work than before because of the program, 59 percent said it supported them in keeping their job, and 54 percent said it allowed them to work more hours. (Policy Studies Associates, Inc., 2004)
After School Alliance: http://www.afterschoolalliance.org/after_out.cfm
28 / 27
Parents’ PerspectivesParents' Perceptions of the Program
1
2
3
4
5
6
7
8
9
10
Feelwelcomed to
visit mychild(ren)
Someone isavailable totalk to me
Feelcomfortableabout asking
aboutprogrammaterials
Staff helps meunderstand
schooldocuments
I am notifiedimmediately ifmy child isn't
payingattention
Staff dealswith my child's
behaviorproblemsquickly
Staff caresabout and
respects mychild
There are staffmembers who
speak myhome
language
Staff showsme ways to
help my childwith
homework
29 / 27
Parents’ Satisfaction
1
2
3
4
5
6
7
8
9
10
The kinds of activitesoffered
The overall performance ofafterschool staff
What my child learns in theafterschool program
The materials and resourcesthe program provides for
parents
Mea
n R
atin
g
30 / 27
Obstacles for Parent Involvement
ASES 60% visited the
program One in three
attended any event One in five
volunteered or given feedback
ASSETs One in five visited
the program One in nine
attended any event One in twenty
volunteered or given feedback
31 / 27
PROGRAM QUALITY
32 / 27
Afterschool Accomplishments
Created a knowledge field that is in general in consensus of program characteristics for high quality indicators
Created a database now possible for investigating longer term effects
Established a “nitch” and demonstrated the significance/importance of ASP in educational research
33 / 27
Common Practices of High Performing Afterschool Programs
Offering a broad array of enrichment activities
Provide a wide range of experiences that promote skill-building and mastery
Intentional relationship-building
Employ strong managers, differentiated Staffing, and
Partner Organizations Provided Support to Project Leaders and Participants
Tasc, 2007
Successful Program Features
Best practices evidenced in the literature on out-of-school time suggest that several critical components such as:
• goal-oriented programs
• program structure
• and program process
These components contribute to the effectiveness and success of programs.
35 / 27
Indicators for High Quality Programs
• Program StructureClearly defined goalsSet up program structures to meet these goalsSet up program mission and vision to motivate staff
• ProcessStrong leadershipHigh quality staffClear communications and supportPositive relationships
• ContentResearch-based curriculum and strategiesBuild in assessment and continuous improvement loop
36 / 27
Theory of Change
Making Programs Accountable and Making Sense of Program Accountability
Outcomes-based
Causal model
Articulate underlying assumptions
37 / 27
Evidence-based Teaching Approach-CAESL
Sequenced - a sequenced set of activities to achieve skill objectives
Active - the use of active forms of learning
Focused – program component to be focused on specific skills
Explicit – the targeting of specific skills
38 / 27
Use Internal Evaluation for Program Improvement
monitoring student progress
measuring program growth
researching program needs
defining areas for professional development
gauging program impact
39 / 27
Model of Data-based Decision Making & Continuous Improvement Process
Analysis of Results
Plan of Action
Specification of
monitoring
Assessment of Student Outcomes
Continuous
Adjustment
Data Based Decision-making
Process
40 / 27
Pyramid of Program Quality – High Scope
Engaged Learning
Encouragement Reframing conflict Skill building Session flow Active engagement Welcoming atmosphere
Psychological safety Physically safe environment Emergency procedures Program space and equipmentHealthy food and drinks
SupportiveEnvironment
Safety
Opportunity for Interaction