Evaluation of After School Programs

Denise Huang

CRESST Conference

September 8th, 2005

What is Evaluation Research?

Evaluation research is the systematic application of social research methods to the assessment of social intervention programs.

It draws upon the concepts and techniques of several disciplines, and is useful at every stage of the programs.

Three Major Focus of Evaluation Research:

Analysis related to the conceptualization and design of interventions

Monitoring of program implementation

Assessment of program effectiveness and efficiency

For Efficient EvaluationDifferent evaluation strategies are called for at

different stages in the life of programs:

In the planning stages of social intervention programs, evaluation focuses on the assessing the extent and severity of the problems requiring intervention, and design programs to alleviate the problems.

In the conduct of new and on-going programs, evaluations help to determine the degree to which the programs are effective.

After the program has been implemented for sufficient time, impact assessments estimate the effects of intervention. For accountability purposes, evaluations consider costs in relation to benefits and compare an intervention’s cost effectiveness.

Strategies for Impact Assessment

Impact assessments are undertaken to determine whether a program has its intended effects

Underlying all impact assessment is the experimental model of comparison

Assessments may make use of quantitative or qualitative data.

Evaluating after school programs: An LA’s BEST Example

The ever evolving role of after school programs:

• Safe haven

• Enrichment activities

• Healthy life style

• Improve academic achievement

LA’BEST: Better Educated students for Tomorrow

• Serving 23,000 students over 133 school sites (147)

• Focus on developing the “whole” child

• Education, enrichment,& recreation

• Research-based curricula

• Community resources

Indicators for program effectiveness

• Student attendance

• Performance measures (achievement tests, homework completion, classroom grades, language re-designation, school retention, future aspirations, etc.)

• Non-cognitive measures (safety, attitudes towards school, relationships with adults, social competence, conflict resolution skills, self-esteem, and self-efficacy etc.)

• Parent involvement

• Long-term effects (drop out rate, life satisfaction, etc)

Challenges in evaluating after school programs

Some of the main concerns:

• Self-selected sample

• Consent forms

• Comparison groups

• High transient rate

• Archive data-record keeping

Encouraging notes

• Dedicated staff

• Happy kids

• Satisfied parents

• Satisfied teachers

• Builds resiliency

Summary of Findings: Academic Performance

Overall LA’s BEST students either maintained or improved their SAT-9 /CAT-6 or CST performances

Overall there is not a difference in academic performance between LA’s BEST and non-LA’s BEST participants

Pre-post analyses indicates that students improved their school attendance after participating in LA’s BEST

Summary of Findings: Social Development

concerns about safety

conflict resolution

academic self-efficacy

study habits

family involvement

future aspirations

Who Benefited Most :

LEP students

Female students

Students who attended more regularly

Students who scored low initially on attitudinal surveys (self-efficacy, work habits, conflict resolution skills)

Students with low performance in test scores

Summary of Findings: Long-term effects

Student and Parent surveys responses between the two groups were very similar regarding:

academic competence

social competence

behavioral indices

future orientations

LA’s BEST parents exhibit a higher degree of parental involvement in school and had higher academic expectations for their children than non-LA’s BEST parents

Former Students’ Perceptions of Positive Effects

Maintain friendships

Former LA’s BEST students

Learn about importance of

school

Develop social and academic

skills

Receive opportunities for mentoring

Parents’ Perceptions of Positive Effects on Child

Parents of former LA’s BEST

students

Improved grades

Improved social skills

Think about college

Implications of the Effects of LA’s BEST After School Program

• Impact of LA’s BEST not limited to academic achievement

• Contributes to improvement in school attendance and social skills, and impacts educational aspirations

• Provides a safety net for students who might otherwise fall through cracks

• Initial findings on drop out rates

Next steps

DOJ long-term effect study

National Partnership study to provide web-based resources to after school programs