Washington 21st Century Community Learning Centers Program

MONTH YEAR

Washington 21st Century

Community Learning Centers

Program Evaluation: Year 2

Neil Naftzger

Matthew Vinson

Feng Liu

Bo Zhu

Kimberly Foley

JANUARY 2014

Washington 21st Century

Community Learning Centers

Program Evaluation: Year 2

January 2014

Neil Naftzger

Matthew Vinson

Feng Liu

Bo Zhu

Kimberly Foley

20 North Wacker Drive, Suite 1231

Chicago, IL 60606-2901

312.288.7600 | Fax: 312.288.7601

www.air.org

Copyright © 2014 American Institutes for Research. All rights reserved. 14-1167_01/14

Contents Page

Executive Summary ......................................................................................................................... i

Program Quality ......................................................................................................................... i

Evaluation Questions and Methods ...........................................................................................v

Methods......................................................................................................................................v

Summary of Key Findings ....................................................................................................... vi

Chapter 1: Introduction ....................................................................................................................1

Evaluation Questions .................................................................................................................1

Reasoning for Chosen Evaluation Questions: Importance of Program Quality ........................1

Organization of Report ..............................................................................................................2

Chapter 2: Methods ..........................................................................................................................3

Data Sources and Analysis.........................................................................................................3

Chapter 3. Primary Characteristics of Washington 21st CCLC Programs and Participants ...........9

Grantee Characteristics ..............................................................................................................9

Center Characteristics ..............................................................................................................12

Summary of Grantee and Center Characteristics .....................................................................24

Chapter 4: Leading Indicators ........................................................................................................25

Overview of Leading Indicators ..............................................................................................25

Selected Leading Indicators .....................................................................................................26

Organization of Leading Indicators Chapter ............................................................................28

Organizational Context ............................................................................................................28

Summary of Organizational Context Findings and Recommendations ...................................36

Instructional Context ................................................................................................................36

Summary of Instructional Context Findings and Recommendations ......................................53

Mutually Reinforcing Context .................................................................................................54

Summary of Findings and Recommendations in Relation to the Community Context

Domain ...................................................................................................................................64

Youth Outcomes Leading Indicators .......................................................................................65

Determining Program Improvement Priorities From the Leading Indicator System .............68

Chapter 5: Assessing 21st CCLC Program Outcomes ...................................................................73

Within-Program and Impact Analyses .....................................................................................73

Within-Program Analyses ........................................................................................................73

Summary of Within-Program Analyses Findings ....................................................................87

Impact of 21st CCLC Participation on Student Achievement .................................................88

Summary of Impact Analyses Results .....................................................................................95

Conclusions ....................................................................................................................................97

References ....................................................................................................................................100

American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—i

Executive Summary

Information summarized in this report is based on data collected and analyzed by American

Institutes for Research (AIR) and the David P. Weikart Center for Youth Program Quality

(Weikart Center) as part of a statewide evaluation of Washington 21st Century Community

Learning Centers (21st CCLC) programs. Results represent findings from Year 2 of a three-year

statewide evaluation. The purpose of this executive summary is to (1) set the context for the

evaluation design with regard to a primary focus on program quality, (2) outline the evaluation

questions and methods, and (3) summarize key findings within each of the identified evaluation

questions. To set the context for the evaluation design, a brief discussion on program quality,

AIR’s framework for understanding afterschool program quality, and the leading indicators of

afterschool program quality developed in collaboration with the Washington Office of

Superintendent of Public Instruction (OSPI) are provided. Following the discussion on program

quality, the evaluation questions and methods are outlined, and a summary of key findings within

each of the identified evaluation questions is presented.

Program Quality

Research on Program Quality

Program quality and the implementation of best practices supported by research are increasingly

recognized as pressing issues for the afterschool field (Granger, Durlak, Yohalem, & Reisner,

2007). Research on the impact of participation in afterschool programming on students’

academic and behavioral outcomes often produces mixed and inconclusive results (Granger,

2008). For example, three noteworthy meta-analyses of the impact of afterschool programming

found that a majority of the reviewed studies did not find better outcomes for afterschool

participants relative to nonparticipants (Durlak & Weissberg, 2007; Granger, 2008; Lauer et al.,

2006; Zief, Lauver, & Maynard, 2006). However, others found average positive effects of

afterschool program participation on students’ academic and behavioral outcomes, which they

attributed to subsets of higher quality programs driving the overall average positive student

outcomes (Durlak & Weissberg, 2007; Lauer et al., 2006). In short, average positive outcomes

across several afterschool programs were likely due to the effectiveness of a small number of

high-quality individual programs. These findings highlight a key relationship between the quality

of afterschool programming and the attainment of desired program outcomes.

Meaningful progress has been made relative to understanding how elements of program quality

support quality afterschool programs and the attainment of desired youth outcomes. For example,

a growing body of research suggests that program outcomes in the form of enhanced student

academic achievement outcomes are realized by delivering developmentally appropriate

programming that is grounded in core principles of youth development (Birmingham, Pechman,

Russell, & Mielke, 2005; Durlak & Weissberg, 2007). The delivery of developmentally

appropriate activities that align with principles of youth development varies as a function of staff

competencies, interpersonal skills, and knowledge (Vandell et al., 2005). Leading experts agree

that staffs’ ability to form meaningful personal staff-student relationships facilitates the delivery

of interactive and engaging program activities (Eccles & Gootman, 2002). Likewise, staff ability

to design and deliver developmentally appropriate and interactive program activities is likely to

American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—ii

differ as a function of the level of support provided by the program overall. As noted by Smith

(2007), Glisson (2007), and Birmingham et al. (2005), a program climate that supports ongoing

staff reflection on and involvement in efforts to improve program quality is a key aspect of

effective youth-development programs. Programs characterized by a supportive and

collaborative climate encourage staff to engage in self-reflective practices that improve overall

program quality.

AIR Framework for Program Quality

The evaluation team at AIR has engaged in extensive work evaluating afterschool programs and

providing technical assistance to support high-quality programming. The framework outlined in

Figure I represents the most recent research related to the path to quality in afterschool programs

as well as the evaluation team’s collective expertise with afterschool programming. As shown in

Figure I, the achievement of desired youth outcomes is a function of complex interactions

between several program elements:

Youth Characteristics. These are the characteristics and contributions youth bring to the

afterschool setting that influence how they engage with and benefit from afterschool

programs.

Community Context. The resources and characteristics of the local and school

community context serve to support meaningful partnerships to develop program goals,

program design, and provide program guidance.

Program Participation. Youth are more likely to benefit from afterschool program

participation if they attend consistently, over a period of time, and participate in a variety

of activity types.

Program Quality. Program quality comprises a series of practices and approaches that

support the provision of developmentally appropriate, high-quality settings and activities

at the point of service. This includes practices and approaches adopted by (a) activity

leaders working directly with youth and (b) the organization as a whole, which provides

an infrastructure to support implementation of effective practice in the design, delivery,

and evaluation of afterschool programming.

American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—iii

Figure I. AIR’s Quality Framework for Afterschool Programs

Leading Indicators of Afterschool Program Quality

To assess the extent to which centers implement research-supported best practices and

approaches, a set of newly defined leading indicators of afterschool program quality was

developed in collaboration with OSPI and the Weikart Center and discussed in the Year 1 report.

The leading indicators are meant to further complement programs’ participation in the Youth

Program Quality Improvement process, to provide additional information regarding how well

programs are progressing in implementing research-supported practices, and more importantly,

to identify areas in need of improvement.

A primary goal of the statewide evaluation was to provide 21st CCLC grantees with data to

inform program improvement efforts regarding their implementation of research-supported best

practices. AIR, the Weikart Center, and OSPI worked collaboratively to define a series of

leading indicators using data collected as part of the statewide evaluation. Specifically, the

leading indicator system was designed to do the following:

Summarize data collected as part of the statewide evaluation in terms of how well the

grantee and its respective centers are adopting research-supported best practices.

Allow grantees to compare their level of performance on leading indicators with similar

programs and statewide averages.

Facilitate internal discussions about areas of program design and delivery that may

warrant additional attention from a program improvement perspective.

American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—iv

The leading indicator system is focused on quality program implementation as opposed to youth or

program outcomes. It is hypothesized and supported by research indicating that more consistent

implementation of research-supported best practices supports the attainment of desired youth

outcomes. During Year 2 of the evaluation, the leading indicators continued to be developed in order

to meet the goals in providing grantees with opportunities for program improvement.

The adopted leading indicators are organized into four overarching contexts: (1) Organizational

Context, which is focused on practices that occur among staff and management; (2) Instructional

Context, which is focused on practices that occur at the point of service, where staff and youth

directly interact; (3) Mutually Reinforcing Context, which is focused on practices related to

coordinating and aligning afterschool programming and activities with the regular school day,

family, and community contexts; and (4) Youth Outcomes Leading Indicators, which are focused

on the change in youths’ proficiency in reading/English language arts (ELA) and mathematics.

Organizational Context

Leading indicators within the Organizational Context examine both staff development and internal

communication and collaboration among program staff. Programs characterized by a supportive

and collaborative climate permit staff to engage in self-reflective practice to improve overall

program quality. Self-reflective practice is more likely to lead to high-quality program sessions that

provide youth with positive and meaningful experiences. Three leading indicators fall under the

Organizational Context: (1) Staff Capacity; (2) Continuous Improvement, which is assessed by

scales measuring program climate and internal communication and collaboration; and (3)

Leadership and Management.

Instructional Context

Leading indicators in the Instructional Context focus on the practices and approaches adopted by

frontline staff to design and deliver activity sessions that intentionally support youth skill

building and mastery that align with centers’ objectives and principles of youth development.

There are two leading indicators in the Instructional Context: (1) Quality of Instructional Content

and (2) Quality of Instructional Processes/Strategies.

Mutually Reinforcing Context

The Mutually Reinforcing Context focuses on relationships between the 21st CCLC program and

context external to the program that significantly impact the success of the program. Community

partners, families, and schools play an important role in the 21st CCLC programs by expanding

program activities, facilitating program sustainability, and providing important information about

student needs. Three leading indicators are associated with the Mutually Reinforcing Context:

(1) Family Engagement, (2) School Context, and (3) Community Context.

Youth Outcomes Leading Indicators

The Youth Outcomes Leading Indicators focus on whether students who regularly attend 21st

CCLC programming (defined as more than 30 days of attendance during the programming

American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—v

period) shifted between state proficiency categories in reading/ELA and mathematics between

the 2010–11 and 2011–12 school years.

Evaluation Questions and Methods

Evaluation Questions

A key objective of the evaluation was to understand how well centers were implementing

research-supported best practices and approaches and to assess the impact of 21st CCLC

participation on students’ academic and behavioral outcomes. Specifically, the evaluation was

designed to answer the following evaluation questions:

1. What were the primary characteristics associated with the grants and centers funded by

21st CCLC and the student population served by the program?

2. To what extent is there evidence that centers funded by 21st CCLC implement research-

supported practices related to quality afterschool programming?

3. To what extent is there evidence of a relationship between center and student

characteristics and the likelihood that students demonstrated better performance on

program attendance and youth outcomes, with a particular emphasis on exploring the

relationship between leading indicator status and these outcomes?

4. To what extent is there evidence that students participating in services and activities

funded by 21st CCLC demonstrated better performance on youth outcomes as compared

with similar students not participating in the program?

Methods

Data Sources

To address the evaluation questions, the evaluation team collected data from the following

sources:

21st CCLC Profile and Performance Information Collection System (PPICS). Data

collected through the Annual Performance Report (APR) module of PPICS on grantee,

center, and student characteristics were extracted from PPICS.

Youth Outcome and Related Data From Comprehensive Education Data and

Research System (CEDARS). Academic and demographic information for 21st CCLC

participants and nonparticipants attending the same schools as 21st CCLC participants

were pulled from the CEDARS database.

Site Coordinator Survey. Site coordinators were surveyed about a variety of program

operations related to implementation of best practices in quality afterschool

programming.

Staff Survey. Program staff were surveyed about a variety of program operations related

to implementation of best practices in quality afterschool programming.

Youth Program Quality Assessment (YQPA) Data. Program staff completed surveys,

self-assessments, and observations as part of a quality assessment improvement program

to support grantees completing the Youth Program Quality Improvement (YPQI) process.

American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—vi

Analysis

Descriptive analysis of PPICS data on grantee, center, and student characteristics along with cluster

analysis techniques were used to provide an overall description of Washington 21st CCLC

operating in the 2011–12 school year. Both descriptive analysis and Rasch analysis of site

coordinator and staff survey responses were used to assess the extent to which centers implement

research-supported best practices aligned with the previously described leading indicator system.

In order to group centers into clusters on the basis of their scores on leading indicator data,

hierarchical cluster analysis was used. Correlational multilevel modeling techniques were employed

to explore the relationship between student- and center-level characteristics associated with 21st

CCLC sites and youth outcomes. Finally, a propensity score matching approach was used to assess

the impact of 21st CCLC programming on youth outcomes by comparing participants with similar

nonparticipants from the same schools.

Summary of Key Findings

A summary of key findings within each of the identified evaluation questions is provided.



Grantee Characteristics

A total of 55 Washington 21st CCLC grantees were active during the 2011–12 school

year.

A majority of grantees (80 percent) were considered “mature” grants—not in the first or

last year of the five-year funding cycle.

Grantees were roughly split between the categories of school-based (53 percent) and non-

school-based (47 percent) grantee.

Center Characteristics

A total of 183 centers were in operation across the 55 active grantees for the 2011–12

school year.

A majority of centers (96 percent) were school based.

Centers mainly served children in elementary school (37 percent) and middle school (34

percent); 14 percent of centers served high school students.

Centers provided an average of 4.4 days of programming a week over eight months.

Roughly half of centers targeted students for enrollment due to students’ low

performance on local or state assessments.

A total of 3,029 staff members worked in centers for the 2011–12 school year.

Centers most commonly employed a mix of mostly school day teachers, other school

staff, and college students (42 percent), and mostly school day teachers (28 percent).

American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—vii

A majority of centers offered mostly enrichment activities (45 percent) or a variety of

activities (27 percent).

A total of 24,379 students attended 21st CCLC programming for at least one day. Of the

total 21st CCLC participants, a majority (61 percent) were regular attendees (attended for

30 days or more).

On average, 21st CCLC regular participants attended 60 days of programming.

Overall, centers had approximately 82 regular attendees and 133 total attendees.

A majority of 21st CCLC participants were Hispanic (45 percent) or white (36 percent).

Most attendees (71 percent) qualified for free or reduced-price lunch, 19 percent were

classified as limited English proficient, and 11 percent were classified as special needs.

2. To what extent is there evidence that centers funded by 21st CCLC implement research-

supported practices related to quality afterschool programming?

As previously noted, leading indicators of afterschool program quality were developed to

examine how well centers implemented research-supported best practices. Findings related to

Evaluation Question 2 are summarized according to the overarching contexts for the leading

indicators and specific leading indicators within each context.


Staff Capacity. Active participation in professional development and training is essential in

supporting staff capacity. Center staff were asked to report on the frequency and type of

professional development/training attended during the 2011–12 school year. A majority of staff

(63 percent) reported participating in some form of training. The most common topic of the

trainings attended by staff included strategies for delivering high-quality academic enrichment

activities and activities to support youth development.

Continuous Improvement. This leading indicator includes the following aspects of continuous

program improvement: (1) program climate, (2) internal communication as reported by site

coordinators, and (3) internal communication as reported by center staff. Key findings within

these aspects of continuous program improvement are summarized below.

Program Climate. The average scale score for program climate reported by center staff fell

within the agree response category (scale response categories included strongly disagree,

disagree, agree, and strongly agree), suggesting that most staff reported supportive,

collaborative program climates. A majority of centers (76 percent) fell in the agree response

category, and 22 percent fell in the disagree or strongly disagree response category. Center staff

were most likely to disagree with the following statements: (1) there is adequate time to plan

individual activity sessions and (2) staff participated fully in program decision making.

Internal Communication—Site Coordinator. For site coordinators, the average scale score for

internal communication fell within the a couple of times per year response category (response

categories included never, a couple of times per year, about once a month, and nearly every

week), suggesting that practices related to internal communication were implemented on an

American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—viii

infrequent basis. A majority of site coordinator scale scores (65 percent) also fell in the a couple

of times per year response category.

Internal Communication—Center Staff. For center staff, average center scale scores fell

within the a couple of times per year response category. However, 27 percent of centers fell

within the a couple of times per year response category, and 57 percent of centers fell in the

about once a month response category. These results suggest that staff are slightly more likely to

engage in strategies for internal communication with other program staff as opposed to engaging

in internal communication strategies with their site coordinators. Center staff were least likely to

report using data to set program improvement goals with other staff, a shift from 2011–12 when

the least frequently implemented activity was observing other afterschool staff delivering

programming.

Leadership and Management. The average scale score reported by staff fell in the three

category (response categories included one, three, and five from the YPQA), suggesting that

most staff reported that the center’s leadership and management support youth-staff relationships

and a positive development focus, promote staff development, and are committed to ongoing

improvement. A majority of centers (77 percent) fell in the three category, and 10 percent fell in

the five response category.


Quality of Instructional Content. Three separate scales were used to assess aspects of

programming related to the quality of instructional content: (1) alignment of program activities

with program objectives, (2) intentionality in program design as reported by site coordinators,

and (3) intentionality in program design as reported by center staff. Key findings within each of

these aspects of quality of instructional content are summarized below.

Alignment of Program Activities With Program Objectives. To assess the extent of alignment

between program objectives and program activities, site coordinators were asked to provide the

top three program objectives, and steps were taken to assess how frequently sites delivered

activities aligned with their top three program objectives. For example, it is expected that

programs identifying improving students’ grade-level proficiency as a top objective would spend

a significant amount of time on academic enrichment activities. From this analysis, results

indicated that a majority of centers (90 percent) delivered activities aligned with their identified

top three program objectives.

Intentionality in Program Design—Site Coordinator. Site coordinators were asked to report

how frequently staff leading program activities engaged in strategies reflective of intentional

program design. Average site coordinator scale scores fell in the frequently response category

(response categories included never, sometimes, frequently, and always), suggesting that site

coordinators felt practices related to intentional service delivery are commonly adopted by

activity leaders. Forty-five percent of site coordinator responses fell in the frequently category.

Intentionality in Program Design—Center Staff. Center staff members were asked to report

how frequently they engage in strategies indicative of intentional program design. The average

center scale score also fell in the frequently response category. A majority of centers (74 percent)

American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—ix

fell in the frequently response category. This indicates that staff are slightly more likely to report

engaging in practices related to intentional program design relative to site coordinator reports of

how frequently staff engage in the same practices.

Quality of Instructional Processes/Strategies

Four separate scales were used to assess aspects of programming related to the quality of

instructional processes/strategies: (1) point of service quality, (2) youth-centered policies and

practices, (3) youth ownership according to the site coordinator survey, and (4) youth ownership

as reported on the staff survey. Key findings within each of these aspects of quality instructional

processes/strategies are summarized below.

Point of Service Quality. The average scale scores for the overall point of service quality fell

within the functioning near optimal category. Percentages of staff respondents stating that point

of service quality, safe environment, and supportive environment were functioning near optimal

were 81 percent, 100 percent, and 89 percent, respectively, although a majority of responses fell

in the still room for improvement category for interaction and engagement.

Youth-Centered Policies and Practices. The average scale scores for this leading indicator fell

within the three category (response options included one, three, and five), with a majority of

centers (76 percent) falling in this category and 10 percent falling in the one category. These

responses suggest that most staff report that programs tap youth interests, build youths’ skills,

and involve youth in the structure and policy of the program.

Youth Ownership—Site Coordinator. The average site coordinator scale score fell in the

disagree response category (response options were strongly disagree, disagree, agree, and

strongly agree). However, in looking at the distribution of site coordinator scale scores, a

majority (51 percent) of site coordinators fell in the agree category. There may be room for

growth in defining more organizational- or state-level strategies for cultivating youth ownership.

Opportunities for Youth Ownership—Center Staff. The average staff survey scale scores fell

in the agree response category. A majority of centers fell in the agree category (60 percent), and

37 percent fell within the disagree category.

Staff Capacity to Create Interactive and Engaging Settings. Staff were asked to rate the

collective capacity of frontline staff to create and provide interactive and engaging program

settings. Average staff scale scores fell in the agree category, suggesting that staff generally

agree that frontline staff adopt strategies likely to produce interactive and engaging settings.

With regard to the distribution of centers across response categories, a majority of centers (70

percent) fell in the agree response category, and 12 percent fell in the disagree category.

Service Delivery Practices. Questions on the service delivery practices scale asked staff to report

on practices they adopt in their own work with youth. The average staff scale score fell in the

available occasionally response category. Likewise, 56 percent of centers fell in the available

occasionally response category.

American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—x


The Mutually Reinforcing Context focuses on relationships between the 21st CCLC program and

context external to the program that significantly impacts the success of the program. Three

leading indicators are associated with the Mutually Reinforcing Context: (1) family engagement,

(2) school context, and (3) community context.

Family Engagement. Survey questions on the site coordinator survey assessed center

approaches to communicating with families. The average family communication scale score fell

within the sometimes response category (response options were never, sometimes, and

frequently), which is indicative of programs typically communicating with families once or twice

a semester. A majority (74 percent) of site coordinator responses fell in the sometimes response

category.

School Context. This leading indicator is meant to capture the degree to which 21st CCLC staff

members align the design and delivery of programming to the school day and individual student

needs. Survey questions related to linkages to the school day and data use were asked on the site

coordinator and staff surveys. The average site coordinator scale score fell within the minor

strategy response category for linkages to the school day, indicating that most sites employed

only a portion of strategies for establishing linkages with the school day. Fifty-three percent of

site coordinator respondents fell within the minor strategy response category, and 32 percent fell

within the major strategy category. For staff responses, the average scale score fell within the

disagree response category, suggesting that, on average, most staff have an incomplete sense of

both student academic needs and school day curriculum and/or instructions. Sixty-five percent of

centers fell in the agree response category, and 31 percent fell in the disagree category. The

average scale score for data use for both site coordinators and staff fell in the occasionally use

category, suggesting that the degree to which they use data is limited. Seventy-three percent of

site coordinators and 55 percent of staff responses fell into this category.

Community Context. The leading indicator for community context is meant to capture the

degree to which partners associated with the center are actively involved in planning, decision

making, evaluating, and supporting program operations, as well as the extent to which the

program adopts practices supportive of family and community engagement. The average site

coordinator scale score on the partner involvement scale fell within the do informally response

category (response options included did not do, do informally, and do formally). Generally,

although centers work with partners in many ways, they have a tendency to do so on an informal

basis as opposed to following formal policies and procedures. Of activities that site coordinators

engage in with partners, 19 percent do so formally, while 60 percent do so informally. It is also

important to note that 112 centers had partners that are actively involved in the provision of

programming directly to youth.

The average scale score for the family and community engagement scale, which is meant to

capture barriers to family and community involvement in the program, fell into the three

category (response options included one, three, and five). Seventy-seven percent of staff

respondents fell into this category, suggesting that a majority of staff reported policies at their

centers that promote family and community engagement in the program.

American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—xi

Youth Outcomes

The leading indicator is meant to capture descriptively the extent to which students participating

in the program moved from one state proficiency category to another between the 2010–11 and

2011–12 school years. For example, 14 percent of regular attendees moved from the well below

standard category to the below standard category in the 2011–12 school year in mathematics.





Correlational relationships between center- and student-level characteristics and youth outcomes

were explored in order to examine whether centers’ leading indicator status was related to

program attendance, academic performance, and unexcused absences. It was hypothesized that

there here would be a negative correlation between center membership in clusters where scores

on all leading indicators were below average and youth outcomes. Key findings include:

Membership in the Instructional Context Content Below Average cluster was negatively

associated with reading scores, as measured through students’ reading state assessments.

Membership in this cluster was also positively associated with unexcused absences. Each

of the findings was consistent with what was hypothesized.

Center membership in the Organizational Context Below Average cluster was negatively

associated with program attendance (this was consistent with what was hypothesized),

while displaying a positive relationship with credits earned and cumulative GPA (which

was not expected).

Membership in the Instructional Context Process Below Average cluster was found to be

significantly and positively related to program attendance. This is surprising but may be

related to the high number of elementary only programs in this cluster, as elementary

programs often have higher attendance than middle or high school programs.

The number of days of attendance in 21st CCLC programs was found to have a positive

relationship with academic performance-related outcomes.


funded by 21st CCLC demonstrated better performance on youth outcomes as

compared with similar students not participating in the program?

Propensity score matching was employed to examine the impact of 21st CCLC programming on

participants as compared to nonparticipants with similar characteristics from the same schools.

Outcomes explored included academic performance and unexcused absences. Key findings are

summarized as follows:

Small but significant positive effects were found for reading and mathematics

achievement at both 30-day and 60-day participation levels when pooled across grades.

Students in the treatment group with 30-day participation achieved 0.027 standard

deviation units higher on reading and 0.044 standard deviation units higher on

mathematics than nonparticipants. For 60-day participation group, 21st CCLC

American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—xii

participants scored 0.033 and 0.035 standard deviation units higher than nonparticipants

on reading and math, respectively.

There was a significant positive impact of the program on the cumulative GPA of

students with 60+ days of treatment; the cumulative GPA of this group was 0.195

standard deviation units higher than the comparison group. A significant positive effect

for this group was also seen for percentage of credits earned.

The participant group showed a statistically significant, negative impact of the 21st

CCLC programming on unexcused absences. Students in both the 30-day and 60-day

treatment groups had unexcused absences of 66 percent and 39 percent of the level in the

nonparticipant group, respectively.

Impacts varied across grade levels. For example, a significant negative impact on

cumulative GPA was found for students in the 60-day treatment group in Grade 9, and

there was a significant positive impact on students in Grades 10 and 11.

Recommendations

This report’s findings on leading indicators, correlational relationships, and impact analyses

provide guidance for grantees on areas for continued growth in the upcoming years, including (1)

using data to inform services for individual students, (2) allowing staff more time for planning and

preparation, and (3) identifying ways to incorporate more youth ownership into the program at

grade-appropriate levels. These results are very similar to those identified in the Year 1 report. In

addition, there appears to be some evidence that (a) there are opportunities for growth in terms of

how centers go about designing and delivering activities from a content perspective and (b) that

enhanced levels of practice in this area are related to better school-related outcomes. Although

OSPI has an infrastructure for supporting instructional quality from a process perspective, it may

want to give consideration to the types of supports it could provide to enhance the manner in which

21st CCLC programs support the cultivation of skills and knowledge from a content perspective,

particularly in relation to the needs of participating youth.

Although a variety of positive program effects were demonstrated in this year, OSPI is interested in

further exploring the types of impacts 21st CCLC is having on social-emotional learning, 21st

century skills and competencies, and noncognitive outcomes. Toward this end, in Year 3 of the

evaluation, AIR will be working to collect information from grantee project directors on what they

believe their programs are impacting in these areas and what their priorities should be in terms

testing measurement strategies to assess program impact on such outcomes. Steps will be taken to

select a sample of instruments designed to measure high-priority outcomes and pilot those in a

small number of centers during spring semester of the 2013–14 school year.

Finally, the leading indicators represent a substantial investment of time and effort to provide

Washington 21st CCLC grantees with actionable data to guide and support program improvement

efforts. A key goal of the Year 3 evaluation will be to better understand the efficacy of these tools

as a vehicle for supporting quality improvement efforts and to highlight portions of the system that

are proven to have especially high value to grantees and OSPI.

American Institutes for Research Washington 21st CCLC Year 2 Evaluation—1

Chapter 1: Introduction

Throughout the past nine years, 21st Century Community Learning Centers (21st CCLC) in the

state of Washington have provided afterschool and expanded learning programming to enhance

the academic well-being of students in high-poverty communities. This report highlights how

well afterschool programs (funded by 21st CCLC, subsequently referred to as centers)

throughout Washington have fared relative to meeting the goals and objectives for supporting

student growth and development as specified by the Washington Office of Superintendent of

Public Instruction (OSPI).

Information discussed in the following sections is based on data collected and analyzed by

American Institutes for Research (AIR) and the David P. Weikart Center for Youth Program

Quality (Weikart Center) as part of a statewide evaluation of Washington 21st CCLC programs.

The results represent findings from Year 2 of a three-year statewide evaluation, which will

conclude in January 2014.

Evaluation Questions

A key objective of the 2011–12 statewide evaluation of Washington 21st CCLC-funded

programming was to understand both how well centers were implementing programming in

terms of research-supported practices and approaches and what impact participation in 21st

CCLC-funded activities had on student academic outcomes. More specifically, the evaluation

was designed to answer the following set of evaluation questions:

1. What were the primary characteristics associated with both centers funded by 21st CCLC

and the student population served by the program?

2. To what extent was there evidence to suggest that centers funded by 21st CCLC had

adopted research-supported practices related to the provision of quality afterschool

programming?








Reasoning for Chosen Evaluation Questions: Importance of Program Quality

Collectively, the domain of evaluation questions represents both the goals and objectives OSPI

has specified for the 21st CCLC program and emerging issues across the national landscape of

afterschool programming. For example, Granger (2008) notes that afterschool research often

demonstrates mixed and inconclusive results regarding the impact of participation in afterschool

programming on students’ academic and behavioral outcomes. Granger (2008) cites three

noteworthy meta-analyses of the impact of afterschool programming that found a majority of the

studies included in each meta-analysis did not find better outcomes for the afterschool participant

group relative to the comparison group (Durlak & Weissberg, 2007; Lauer et al., 2006; Zief,


Lauver, & Maynard, 2006). However, both Durlak & Weissberg (2007) and Lauer et al. (2006)

found average positive effects of participation in afterschool programming on students’ academic

and nonacademic outcomes, which they attributed to higher quality programs driving the overall

positive outcomes. In short, average positive outcomes across several afterschool programs were

likely due to the effectiveness of a small number of high-quality individual programs. These

findings highlight a key relationship between the quality of afterschool programming and the

attainment of desired program outcomes.

Although meaningful progress has been made in understanding elements of quality in afterschool

programming (e.g., Granger, Durlak, Yohalem, & Reisner, 2007; Little, 2007; Vandell et al.,

2005; Wilson-Ahlstrom & Yohalem, 2007; Yohalem, Wilson-Ahlstrom, Fischer, & Shinn,

2009), these understandings have largely been used to support the development of quality

assessment tools and improvement systems to help afterschool programs better understand (1)

criteria for afterschool program quality, (2) how well they measure up to identified criteria, and

(3) steps that can be taken to modify programming and enhance program quality. This reflects

the stance of leading researchers that the most pressing issue before the afterschool community is

developing effective quality-improvement systems (Granger et al., 2007).

OSPI, in collaboration with the Weikart Center, has taken steps to craft a quality assessment

improvement system and support grantees in completing the Youth Program Quality

Improvement (YPQI) process, which utilizes a youth development framework to combine self-

assessment, action planning, skill development, and targeted technical assistance to enhance

program quality. To address Evaluation Question 2 as noted previously, this year’s report

summarizes and expands the leading indicators introduced in the Year 1 report. The leading

indicators developed as part of the statewide evaluation are meant to further complement

programs’ participation in the YPQI process, to provide additional information regarding how

well they are progressing in implementing research-supported practices, and more importantly,

to identify areas of program operations in need of improvement. OSPI’s use of the YPQI process

and leading indicators provides 21st CCLC programs with an infrastructure to make data-driven

decisions about program improvement in a timely, meaningful, and systematic way. As

indicated, one of the goals of the statewide evaluation is to explore the relationship between

measures of program quality, as measured by the leading indicators, and student academic and

behavioral outcomes. Exploring this relationship is especially helpful in refining the leading

indicator system according to measures of program quality that relate to student outcomes.

Organization of Report

The following sections provide a summary of the methods, including data sources and analytic

techniques, to address the primary evaluation questions. Following an overview of the evaluation

methods, key grantee and center characteristics are summarized, with a particular emphasis on

characteristics that are considered key to improving student academic achievement and attaining

desired program outcomes. The leading indicator system is then summarized and explained with

regard to how information relates to future evaluation and technical assistance efforts. Finally,

analyses for assessing relationships between center- and student-level characteristics and student

outcomes, as well as for evaluating the impact of 21st CCLC participation on student-level

outcomes are summarized, along with conclusions and recommendations to guide future

evaluation and program improvement efforts.


Chapter 2: Methods

Data Sources and Analysis

Data collected and analyzed in this report come from four primary sources, including

administrative data systems and surveys. Each data source and associated methods of data

analysis are described.

21st CCLC Profile and Performance Information Collection System (PPICS)

PPICS is a Web-based data collection system developed and maintained by AIR on behalf of the

U.S. Department of Education. Data on the full domain of 21st CCLC programs funded

nationally, including those in Washington, are collected through this system. Data collected

through the Annual Performance Report (APR) module of PPICS on center characteristics in

relation to the 2011–12 programming period were extracted from PPICS and utilized in several

analyses contained in this report, including information on program operations, staffing,

activities provision, and student attendance rates. A total of 184 programs associated with 55

active 21st CCLC grantees, during the 2011–12 programming period, were represented in the

data set extracted from PPICS. (Note: A single 21st CCLC grant typically has more than one

program associated with it.)

Youth Outcome and Related Data From CEDARS

AIR constructed a unique data collection module for Washington integrated within PPICS that

allowed for the collection of student-identifiable information that was extracted from the system

and provided to OSPI. OSPI used this information to perform a series of merges against state data

warehouses to obtain Measurements of Student Progress (MSP) reading and mathematics scores,

High School Proficiency Exam (HSPE) reading scores, cumulative GPA, credits earned, and the

number of unexcused absences, as well as additional demographic information about the students

in question from the Comprehensive Education Data and Research System (CEDARS), a

longitudinal data warehouse of educational data maintained by OSPI. OSPI also identified students

not participating in 21st CCLC programming who attended the same schools as 21st CCLC

participants and provided the same testing and related CEDARS information for these students.

These data were used both to conduct the impact analyses predicated on comparing 21st CCLC

participant with nonparticipant outcomes and to construct the set of models needed to explore the

relationship between center and student characteristics and student achievement and related

outcomes.

Site Coordinator Survey

An online survey of site coordinators working in 21st CCLC programs active during the 2011–12

school year was administered in spring 2012. The site coordinator was defined as the individual

at a given center who was responsible for the day-to-day operations of the program and was the

initial point of contact for parents and staff when questions or issues arose on-site. Generally, site

coordinators are seen as important middle managers in the delivery of 21st CCLC programming

at a given site.


A total of 184 site coordinator surveys were administered. Completed surveys were received

from 173 site coordinators, for a response rate of 94 percent. The survey addressed the extent to

which centers engaged in practices that the research indicates are supportive of effective

afterschool programming. Sets of survey questions were organized to create scales measuring the

following dimensions of program operations:

Program objectives

Activity enrollment policies and recruitment approaches

Access to and use of student data

Linkages to the school day

Staffing approach and challenges

Other operational challenges

Intentionality in activity and session design

Creation of interactive and engaging settings for youth

Opportunities for youth ownership

Internal communication designed to support program development and improvement

Practices supportive of cultivating effective partnerships

Practices supportive of parent involvement and engagement

Professional development and training

Data obtained from the site coordinator surveys were used both to support the leading indicator

process and to construct variables included in analyses to assess the relationship between center

characteristics and 21st CCLC participant outcomes.

Staff Survey

The purpose of the online staff survey was to obtain information from frontline staff who worked

directly with youth during the 2011–12 school year. A particular focus of the survey was on

practices that support both positive academic outcomes and youth development outcomes. As

with the site coordinator survey, the staff survey included sets of questions associated with a

given scale, as well as open-ended questions to assess dimensions of program operations.

Dimensions of program operations assessed on the staff survey included the following:

Program objectives

Intentionality in activity and session design

Practices supportive of academic skill building, including linkages to the school day and

using data on student academic achievement to inform programming

Practices supportive of positive youth development

Opportunities for youth ownership

Internal communication designed to support program development and improvement


Program climate in terms of how staff view the organizational supports and structures as

supporting their work with youth

Training participation

Completed surveys were received from 1,090 center staff from 181 centers. The number of

completed staff surveys received per center ranged from one to 19, with an average of six

completed surveys per center. As with the site coordinator survey, data obtained from the staff

surveys were used to support the leading indicator process.

Youth Program Quality Assessment (YPQA) Data

As noted previously, OSPI, in collaboration with the Weikart Center, has taken steps to craft a

quality assessment improvement system and support grantees in completing the YPQI process.

As part of this process, observations were conducted by program staff as a self-assessment or by

trained external observers of activities provided by 21st CCLC grantees, and the YPQA Form A

was scored to provide an estimate of how safe, supportive, interactive, and engaging the

observed session was for participating youth. In addition, although the YPQA Form A is meant

to measure program quality at the point of service, the YPQA Form B is a rubric completed by

program staff on how well the program has adopted organizational processes that are likely to

engender and facilitate point of service quality. Both YPQA Form A and B data were uploaded

directly to the Weikart Center via the Center’s online score reporter.

It is important to note that participation in the YPQI process was voluntary for Washington 21st

CCLC grantees during the 2011–12 school year. As a result, PQA Form A data were available

for only 72 centers associated with 33 grantees. Form B was provided in relation to 62 centers

associated with 32 grantees.

Analytic Approach and Methods

It is important to note that the findings outlined in this report are primarily quantitative in nature.

This approach was largely driven by both the evaluation questions being answered and the

resources available to carry out the project during Year 2 of the project. Analyses highlighted in

this report fall within five general categories:

1. Descriptive Analyses. Information related to grantee, center, and student characteristics

obtained from PPICS, the surveys, and the PQA were analyzed descriptively to explore

the range of variation on a given characteristic. Some of the leading indicators also were

calculated employing descriptive analysis techniques.

2. Analyses to Create Scale Scores. Many questions appearing on the site coordinator and

staff surveys underpinning the leading indicators were part of a series of questions

designed to assess an underlying construct/concept, resulting in a single scale score

summarizing performance on a given area of practice or facet of 21st CCLC afterschool

implementation (e.g., practices that support linkages to the school day). An example is

shown Figure 1, which outlines the questions making up the Intentionality Program

Design scale that appeared on the site coordinator survey.


Figure 1. An Example of a Survey Scale Calibrated Using Rasch Techniques

For scales such as this, Rasch scale scores were created using staff and site coordinator

responses to a series of questions to create one overall score. These scale scores ranged

from 0 to 100, where higher scores were indicative of a higher level or more frequent

adoption of a specific quality practice or set of practices.

Scale scores resulting from the application of Rasch approaches can also be used to

classify what portion of the rating scale the average scale score fell within. For example,

the statewide mean value for the Intentionality in Program Design scale highlighted in

Figure 1 was 59.82, which put the statewide average in the frequently range of the scale

indicating the typical staff member responding to the survey reported engaging in these

practices on a frequent basis. This approach also allowed the evaluation team to explore

the distribution of centers in light of what response option their average scale score put

them in.

The primary benefit of this approach is the capacity to distill responses from several

questions down into one overall score for the center, simplifying the process of

interpreting how a center did on a given element of quality, particularly in relation to

other programs in the state.

3. Hierarchical Cluster Analysis. Hierarchical cluster analysis was employed to combine

centers into groups based on how well they scored on the leading indicator data collected


during the 2011–12 school year. Cluster analysis is typically employed to combine cases

(or, in this case, centers) into groups using a series of variables as criteria to determine

the degree of similarity between individual cases and is particularly well suited when

there is a desire to classify a large number of cases into a smaller domain of discrete

groupings. Employing this approach allowed the evaluation team to synthesize the full

domain of leading indicator data into a series of more discrete and meaningful quality

profile types, making it easier to describe how centers active during the 2011–12 school

year performed relative to the indicators overall and to create variables that could more

easily be added to the multilevel models described below.

4. Correlational Multilevel Modeling Techniques. Several multilevel models were run to

explore the relationship between center-level and student-level characteristics associated

with sites funded by 21st CCLC and student-level outcomes, including attendance in 21st

CCLC programs and performance on state assessments in reading and mathematics and

other school-related outcomes. Although these analyses afford the capacity to say if a

significant relationship existed between a center- or student-level characteristic and a

given outcome such as mathematics achievement, these approaches cannot indicate that a

given characteristic caused a given outcome. In this sense, these analyses are

correlational, but not causal, in nature.

5. Propensity Score Matching. In contrast to the multilevel modeling techniques,

propensity score matching approaches were employed to estimate the causal impact of

21st CCLC participation on student performance in reading and mathematics using MSP

and HSPE scores obtained from OSPI, as well as a series of other school-related

outcomes. Given that 21st CCLC program participants were not randomly assigned to

participate in the program, the problem of selection bias was an issue that needed to be

addressed before program impact could be explored from a causal perspective. It is likely

that students who participated in 21st CCLC programming were different from those

students attending the same schools who do not enroll in 21st CCLC. These differences

can bias estimates of program effectiveness because they make it difficult to disentangle

preexisting differences between participants and nonparticipants from program impact.

Propensity score matching was used to mitigate that existing selection bias in program

effect.

Table 1 provides a summary of the methods that were employed to answer each evaluation

question.


Table 1. Summary of Methods by Evaluation Question

Evaluation Question Descriptive

Analysis

Rasch

Analysis

Hierarchical

Cluster

Analysis

Correlational

Multilevel

Modeling

Propensity

Score

Matching

What were the primary characteristics associated with

the grants and centers funded by 21st CCLC and the

student population served by the program?

To what extent was there evidence that centers funded

by 21st CCLC implement research-supported

practices related to quality afterschool programming?

To what extent is there evidence of a relationship

between center and student characteristics and the

likelihood that students demonstrated better

performance on program attendance and youth

outcomes, with a particular emphasis on exploring the

relationship between leading indicator status and these

outcomes?

To what extent is there evidence that students

participating in services and activities funded by 21st

CCLC demonstrated better performance on youth

outcomes as compared with similar students not

participating in the program?


Chapter 3. Primary Characteristics of Washington 21st

CCLC Programs and Participants

One of the hallmarks of the 21st CCLC program is the wide diversity (1) of organizations

involved in the provision of 21st CCLC programming, (2) of approaches to the manner in

which services and activities are delivered, and (3) in the nature of the student population

served. In this chapter, the primary characteristics associated with both grantees and centers

funded by 21st CCLC and the student population served by the program are outlined in relation

to the 2011–12 programming period.

Grantee Characteristics

OSPI is responsible for distributing 21st CCLC funds it receives from the U.S. Department of

Education via a competitive bidding process that results in the awarding of new grants to

entities that propose to operate centers in high-poverty communities. Grants active during the

2011–12 programming period were initially awarded in 2007 (n = 11), 2008 (n = 12), 2009

(n = 21), and 2010 (n =11). (No grants were reported with an award date in 2011.) The term

grantee in this report refers to an entity that applied for and received a 21st CCLC grant from

OSPI, serving as the fiscal agent for the grant in question. This section considers elements that

can be considered only at the grant level, notably grant maturity, grant organization type, and

first-year award amounts. Where feasible, an effort has been made to compare Washington

grantees with all grantees nationwide active during the 2011–12 reporting period tracked in

PPICS.

Grantee Maturity

Grantee maturity was examined as part of the evaluation because of the following hypothesis:

Based on their experience, more mature centers have found ways to provide higher quality

services, adapt more readily to budget reductions, and have planned to sustain the programs after

the grant funding ends. To facilitate comparisons with national data housed in PPICS,

Washington grantees were classified into three possible maturity categories:

New: grantees in their first year of 21st CCLC funding

Mature: grantees not in their first year, but also not in their last year of funding

Sustaining: grantees in their last year of 21st CCLC funding

As shown in Table 2, among Washington grantees active during the 2011–12 programming

period, the vast majority were found to fall in the mature category (80 percent), and the

remaining grants were in their last year of operation (20 percent). Grants were given for a five-

year period.


Table 2. Grants by Maturity

WA Grants

All Grants Nationwide*

Grant Maturity

N Grants

% Grants

N Grants**

% Grants

New 0 0.0% 421 10.6%

Mature 44 80.0% 2,072 52.1%

Sustaining 11 20.0% 1,974 37.3%

Total Grantees 55 100.0% 3,974 100.0%

*Note. As stated in Chapter 2, the national numbers were not finalized at the time of compiling this report;

four states were still incomplete. These numbers therefore reflect the vast majority of grantees nationwide,

but not all.

**Organization maturity could not be determined for 142 grantees at the national level.

Grantee Organization Type

As established in the authorizing legislation for 21st CCLC, programs may be administered by

several types of grantee agencies. The most relevant distinction is whether or not the grantee

organization is a school-based entity. School-based organizations (SBO) include school districts,

charter schools, and private schools. Non-school-based organizations (NSBO) include, among

other entities, community-based organizations, faith-based organizations, health-based

organizations, and park districts.

Of the 21st CCLC grantees funded by Washington, school-based and non-school-based

organizations have been represented roughly equally since the state-administered program began.

During the course of the 2011–12 programming period, for example, school districts were the

fiscal agents on 29 of the 55 active grants (53 percent of all 21st CCLC grants). Figure 2 shows

the comparison across six APR years.


Figure 2. School-Based Versus Non-School-Based Grantees

Of the non-school-based grantees, Regional/Intermediate Educational Agencies are the largest

group, making up more than 18 percent of all grantees in 2012, substantially higher than what is

the case nationwide. The next highest non-school-based grantee type was community-based

organizations, making up approximately 16 percent of all fiscal agents, which is comparable with

national norms.

Grant Amounts

Washington’s first-year grant award amounts and the duration of the grants were assessed

alongside national averages, as shown in Table 3. No major differences in terms of the average

length of a grant were noted between the two groups, although the average first-year award for

Washington grantees was somewhat lower than the national average. The median first-year

award amounts for Washington and the nation (Washington inclusive) were, respectively,

$220,000 and $200,000, indicating a smaller number of very large grants is driving the national

average to a higher amount.

13 13 17

20 21 26 26

16 16 14

16

25

29 29

0

10

20

30

40

50

60

2006 2007 2008 2009 2010 2011 2012

School Based

Non-School Based


Table 3. Grants by First-Year Award Amount*

WA Grants

All Grants Nationwide

Award Amount and Duration

Mean

Mean

Year 1 award amount $271,259 $329,451

Award length 5 4.5

Total grantees

Mean number of centers per grant

55

3.3

4,116

2.5**

*Of grantees reporting data for APR 2012.

**Exclusive Washington Grants.

Center Characteristics

One of the primary goals of this report is to examine the relationship between key center

characteristics and the likelihood that centers will have a positive impact on student achievement

and behavioral outcomes. It is important to note that in this report, the term center is used to refer

to the physical location where 21st CCLC-funded services and activities take place. Centers are

characterized by defined hours of operation, have dedicated staffs, and usually have positions

akin to site coordinators. Each 21st CCLC grantee in Washington has at least one center; many

grantees have more than one center. During the course of the 2011–12 reporting period, there

were a total of 183 centers providing 21st CCLC-funded activities and services.

In addition, center characteristics can be termed either to be indicative of research-supported best

practices or simply as innate attributes of the center in question without a strong connection to

the afterschool quality practice literature. Center characteristics indicative of the latter might

include the grade level served, program maturity, and organizational type. For example,

identifying a program as one that serves only elementary students says nothing about the quality

of that program. Although these types of variables are included in models oriented toward

assessing the impact of the program on desired student outcomes, this report does not focus on

them in depth.

Other characteristics, such as the activity (e.g., mostly tutoring, mostly academic enrichment)

and staffing model employed, at a site are still somewhat ambiguous when viewed from a quality

practice standpoint, with the literature less clear on the superiority of certain activities or staffing

approaches. Some preliminary results derived from the PPICS data set seem to show certain

advantages in these areas (i.e., mostly tutoring programs and program staffed by school day

teachers), but the manner in which these data are collected and processed do not lend themselves

to robust casual inferences about the viability of one approach instead of another. Similar

analyses conducted as part of other statewide evaluations have produced more ambiguous results

in terms of how these characteristics may be related to student outcomes. The analyses contained

in this report are intended to build an understanding of whether certain activity or staffing

models seem to be more often correlated with positive youth outcomes and thereby warrant

consideration as a quality practice worthy of emulation and replication. As with the


characteristics detailed earlier, however, this report does not spend a great deal of time exploring

them from a purely characteristic standpoint.

Finally, the domain of characteristics assessed through the site coordinator and staff surveys are

meant to clearly reflect the best-practices literature. Particular attention will be dedicated in this

report to explaining how staff responded to site coordinator and staff survey questions and what

this response may mean in terms of how programs design and deliver activities in ways that are

consistent with best practices. These results are highlighted in particular in the section dedicated

to explaining the newly adopted leading indicator system.

Center Organization Type

Just as with grants, centers can be classified as either school-based or non-school-based. During

the 2011–12 programming period, approximately 95.6 percent of Washington’s centers were

located in schools. This percentage is a little more than the national average of 86.7 percent.

Figure 3. School-Based Versus Non-School-Based Centers

School-Year and Summer Operations

In terms of periods of operation, Washington centers tended most often to offer programming

after the school day (as opposed to before the school day, during the school day, or on

weekends), offering on average 10.3 hours of programming after school each week. On average,

Washington offered slightly less programming during the school year than did centers across the

nation, with roughly 12.1 hours of programming per week compared with a national average of

13.4 hours per week. Washington centers offered programming an average of 4.4 days per week

over 32 weeks, which is similar to the national averages for the 2011–12 programming period.

152 145 142

155 165

181 175

6 4 2 5 7 4 8

0

20

40

60

80

100

120

140

160

180

200

2006 2007 8 2009 2010 2011 2012

School Based

Non-School Based


In terms of summer operations, a total of 121 of Washington’s centers (66.1 percent) offered

summer programming. This was an increase from previous years: The percentage of centers with

summer programs was 55.1 percent in 2006, 62.4 percent in 2007, 45.1 percent in 2008, 48.8

percent in 2009, 34.3 percent in 2010, and 59.5 percent in 2011. In this regard, in 2012

Washington centers were slightly more likely than other centers nationwide to offer summer

programming (with a national average of 53.9 percent). Otherwise, Washington centers tended to

be very similar to other centers nationwide in terms of summer operation averages. Washington

centers with summer programs had, on average, 5.0 weeks of programming (compared with 5.3

weeks nationally) and approximately 19 hours of programming per week (compared with 25

hours of programming per week nationally). Overall, Washington centers are fairly typical for

the nation in terms of program operation.

Center Staffing

The quality of center staffing is crucial to the success of afterschool programming (Vandell et al.,

2005), and many of the program improvement approaches being used in the field emphasize the

importance of staff for creating positive developmental settings for youth. The success of

afterschool programs is critically dependent on students forming personal connections with the

staff—especially for programs serving older students, where a much wider spectrum of activities

and options is available to youth (Eccles & Gootman, 2002).

Similar to their counterparts nationally, Washington 21st CCLC programs employ a variety of

staff, including academic teachers, nonacademic teachers, college and high school students,

counselors, paraprofessionals from the school day, youth development workers, and other

program staff with a wide spectrum of backgrounds and training. A total of 3,029 staff members

were reported for 2011–12 school year operations (33.6 percent volunteer) and 1,081 for the

summer of 2011 (31.2 percent volunteer). Of the school year staff, 24.9 percent were paid school

day teachers. Another 13.0 percent were paid staff with a college degree. Volunteer high school

students were the largest volunteer group, accounting for 9.9 percent of school year staff.

Summer staffing was very similar to school year staffing in terms of relative configuration, with

28.7 percent of summer staff being paid school day teachers, and 11.0 percent being other paid

staff with a college degree. Volunteer community members accounted for 6.8 percent of all

summer staff.

To summarize the different staffing models used by programs active during the 2011–12

programming period, centers were classified into groups or clusters based on the extent to which

they relied on different types of staff to deliver activities, using cluster analysis techniques.1 Data

used to construct these clusters were obtained from PPICS. Figure 4 presents the five primary

staffing models that were identified in the programs.

Based on this analysis, Washington has a relatively high percentage of centers classified as

(a) Mostly School Day Teachers and Other School Staff as well as (b) Mostly School Day

Teachers, the two most common staffing clusters at the national level.

1 Cluster analysis is typically employed to combine cases into groups using a series of variables as criteria to

determine the degree of similarity between individual cases. Cluster analysis is particularly well suited to a study in

which there is a desire to classify a large number of cases into a smaller domain of discrete groupings.


Figure 4. Staffing Clusters, Washington and the Nation (2012 APR)

Note. Based on 183 centers in Washington and 9,560 centers nationally with complete staffing information.

Center Activities

Both the staff working at a given 21st CCLC and the activities offered to students attending the

program in question are critical elements in how youth experience and potentially benefit from

their participation in 21st CCLC. Nationally, the goal of the 21st CCLC program is to provide

academic and nonacademic enrichment programs that reinforce and complement the regular

academic program of participating students. This overarching charge is broad and encompasses a

host of different types of activities, including the following types that are tracked in PPICS:

Academic enrichment learning program

Recreational activity

Homework help

Supplemental Education Services tutoring

Activity to promote youth leadership

Expanded library service hours

Drug/violence prevention, counseling, or character education

Career/job training

Promotion of family literacy

Mentoring

Community service/service learning

17.3%

38.2%

2.6%

30.7%

11.1%

16.7%

27.8%

2.2%

42.2%

11.1%

0%

10%

20%

30%

40%

50%

60%

70%

YD, Oth No Coll,

SD Teach

SD Teach Oth, SD Teach SD Teach, Oth

School Staff

College Stu, SD

Teach

All States

Washington


Promotion of parent involvement

Other (e.g., activities involving computers and technology, life skills, nutrition, etc.)

In order to further classify centers into categories that meaningfully represent the relative

emphasis given to providing different types of activities (academic enrichment, tutoring,

homework help, recreation, etc.), K-Means cluster analysis also was employed using center-level

percentages for each category of activity. When compared with the nation, centers in

Washington were more likely to fall into the Enrichment cluster (45 percent of all centers

compared with 23 percent of centers nationally) or the Variety cluster (with 27 percent of all

centers in Washington, compared with 34 percent nationally). See Figure 5.

Figure 5. Activity Clusters, Washington and the Nation (2011–12 Programming Period)

Note. States have the option to require their centers to submit activities data in the APR in one of two

different ways: as aggregated hours or as individual activity records. Because only individual activity

records are used to carry out the cluster analysis in question, the numbers presented under “Activity

Cluster” represent centers in states that opted to employ the individual activity record option. For all

states, there were 4,541 centers with individual activity cluster designations (Washington inclusive); for

Washington, there were 155 centers with individual activity cluster designations.

Grade Levels Served

A topic garnering increasing attention at the national level relates to the role that grade level

plays, both in terms of how 21st CCLC programs should structure their operations and program

activities and the outcomes for which they should be accountable through performance indicator

systems. Using student-level data about the grade level of students attending a program, 21st

CCLC programs were classified as follows:

Elementary Only: Centers serving students up to Grade 6

23.9%

9.4%

33.8%

20.7%

12.2%

7.7% 10.3%

26.5%

45.2%

10.3%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

Recreation Tutoring Variety Enrichment Homework Help

All States

Washington


Elementary/Middle School: Centers serving students up to Grade 8

Middle School Only: Centers serving students in Grades 5–8

High School Only: Centers serving students in Grades 9–12

Other: Centers that did not fit one of the other five categories

The High School Only category is especially important to examine because afterschool programs

for older children often look considerably different from elementary or middle school programs

(Naftzger et al., 2007). High school students’ needs are different from younger students, and they

often have other afternoon obligations such as jobs or extracurricular activities. In terms of grade

levels served, centers in Washington most commonly served elementary school students

exclusively, with 37 percent of all centers being classified as Elementary Only during the 2011–

12 programming period. However, as Figure 6 shows, starting during the 2008–09 programming

period, centers serving middle school age youth became increasingly common, representative of

an OPSI-initiated policy shift to fund more programs serving middle and high school age youth.

Figure 6. Percentage of Centers per Grade-Level Cluster, per Year

Note. Reflective of 183 centers with grade-levels-served status available.

Center Attendance

Attendance is an intermediate outcome indicator that reflects the potential breadth and depth of

exposure to afterschool programming. In this regard, attendance can be considered in terms of

(1) the total number of students who participated in the center’s programming throughout the

course of the year and (2) the frequency and intensity with which students attended programming

when it was offered. The former number can be utilized as a measure of the breadth of a center’s

reach, and the latter can be construed as a measure of how successful the center was in retaining

students in center-provided services and activities.

54

%

9%

14

%

4%

4%

14

%

57

%

11

%

15

%

4%

4%

9%

47

%

11

%

26

%

6%

6%

5%

38

%

7%

36

%

4%

10

%

5%

38

%

4%

35

%

6%

15

%

2%

38

%

4%

35

%

6%

15

%

2%

37

%

6%

34

%

8%

14

%

2%

0%

10%

20%

30%

40%

50%

60%

70%

Elem Elem-Mid Mid Mid-High High Other

2006 2007 2008 2009 2010 2011 2012


As part of the APR data-collection process in PPICS, information was collected on the total

number of students that a given center served during the programming period; how many of

those students met the definition of regular attendee by participating in 30 or more days of

programming; and demographic information about the student population in question, including

grade level and ethnicity.

In Washington, a total of 24,379 students were reported as attending 21st CCLCs for at least one

day during the 2011–12 programming period. Of these, 14,966 were regular attendees (students

who attended a total of 30 days or more during the reporting period), or 61.4 percent (compared

with 49.9 percent nationally). Attendance levels year-over-year are presented in Figure 7. The

decline in attendance levels between 2009 and 2010 is representative of an OSPI-adopted policy

change that increased the number of days a student would need to attend in order to be counted

as a participant.

Figure 7. Attendees and Regular Attendees in Washington State, by APR Year

Nearly half of students who met the definition of regular attendee participated in 21st CCLC-

funded activities for 30 to 39 days, with a steady decline in the number of students attending with

each increasing 10-day attendance band. See Figure 8.

9,708 9,167 9,426 12,304

13,877 14,951 14,966

13,502 13,276 10,022

15,627

6,608

8,829 9,413

0

5,000

10,000

15,000

20,000

25,000

30,000

2006 2007 2008 2009 2010 2011 2012

Non-Regular Attendees (Students)

Regular Attendees (Students)


Figure 8. Number of Students by Number of Days Attended

Overall, the mean school year attendance rate for regular attendees was 60 days, with a median

of 53. For summer, the mean attendance rate for regular attendees was 15 days, with a median of

14 days. On average, each center in Washington had approximately 133 total students and 82

regular attendees. This was about the same as total attendance in APR 2011. Median values show

a similar trend. See Figure 9 for year-over-year trends.

Figure 9. Average Attendance Rate per Center by APR Year,

Total and Regular Attendees (Washington Only)

In terms of ethnicity, Washington centers mostly served Hispanic and white students, with 45

percent of all regular attendees being Hispanic and 36 percent of regular attendees identified as

4,213

2,290

1,876 1,586

1,203 978

847 650

486 307

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

30-39 40-49 50-59 60-69 70-79 80-89 90-99 100-109 110-119 >= 120

To

tal

Nu

mb

er o

f R

egu

lar

Att

end

ees

Total Days Attended (Summer 2011 and School Year 2012)

146.9 150.6

135.1

174.6

119.1 128.5 133.2

61.4 61.5 65.5 76.9 80.7 80.8 81.8

0

20

40

60

80

100

120

140

160

180

200

2006 2007 2008 2009 2010 2011 2012

Total Students (Avg)

Regular Attendees (Avg)


being white. See Figure 10 for more detail on the number of students served in Washington by

ethnic group.

Figure 10. Number of Total Students and Regular Attendees, by Ethnicity

The 21st CCLC facilities have been specifically designed to provide afterschool activities and

services to students living in high-poverty communities. Typically, student eligibility for free and

reduced-price lunch is the metric relied upon to assess how well states and grantees are reaching

this target population. As shown in Figure 11, roughly 71 percent of all attendees and 75 percent

of regular attendees were eligible for free and reduced-price lunch (FRPL) during the 2011–12

programming period.

Figure 11. Number of Total and Regular Attendees, by FRPL Status

Note. The number of students whose FRPL status was unknown is not shown.

10,745

8,894

1,468 1,318 1,409

6,616

5,293

997 827 917

0

2,000

4,000

6,000

8,000

10,000

12,000

Hispanic White Black Asian Native American

Total Attendees

Regular Attendees

23,210 22,443

19,448

27,931

20,485

23,780 24,379

9,708 9,167 9,426

12,304 13,877

14,951 14,966

12,917 15,016 13,233 18,490 13,801 16,133 17,301 5,269 6,615 7,034 9,107 10,081 10,675 11,196 0

5,000

10,000

15,000

20,000

25,000

30,000

2006 2007 2008 2009 2010 2011 2012 2006 2007 2008 2009 2010 2011 2012

Total Attendees Regular Attendees

Total Students

FRPL Unknown

FRPL


In addition to FRPL eligibility, additional information about the student population served by

21st CCLC recorded in PPICS includes students designated as being limited English proficient

(LEP) and as having special needs. In 2011–12, 19 percent of all students and 21 percent of

regular attendees were LEP students, and 11 percent of all attendees and 11 percent of regular

attendees were classified as having a special need of some sort. Additional information about

each of these subgroups is outlined in Figures 12 and 13.

Figure 12. Number of Total and Regular Attendees, by Limited-English-Proficiency Status

Note. The number of students whose LEP status was unknown is not shown.

Figure 13. Number of Total and Regular Attendees, by Special-Needs Status

Note. The number of students whose special-needs status was unknown is not shown.

23,210 22,443

19,448

27,931

20,485

23,780 24,379

9,708 9,167 9,426

12,304 13,877

14,951 14,966

3,809 4,636 4,150 4,896 3,474 3,824 4,555 1,670 2,194 2,221 2,817 2,746 2,853 3,143 0

5,000

10,000

15,000

20,000

25,000

30,000

2006 2007 2008 2009 2010 2011 2012 2006 2007 2008 2009 2010 2011 2012


Total Students

LEP Unknown

LEP

23,210 22,443

19,448

27,931

20,485

23,780 24,379

9,708 9,167 9,426

12,304 13,877

14,951 14,966

1,373 1,936 1,774 2,669 2,117 2,457 2,605 645 813 955 1,295 1,437 1,683 1,690 0

5,000

10,000

15,000

20,000

25,000

30,000

2006 2007 2008 2009 2010 2011 2012 2006 2007 2008 2009 2010 2011 2012


Total Students

Spec Needs Unknown

Spec Needs


Enrollment Policies and Recruitment Approaches

Enrollment policies and recruitment practices may have a substantial bearing on program design

and delivery. For example, a program that targets a relatively small number of students with high

academic needs and proposes to provide them with intensive support in one-on-one and small-

group settings will have different strategies for recruitment and enrollment from a program that

aims to serve as many students as possible and provide those students with a rich array of

academic and nonacademic enrichment activities. Questions related to each of these areas were

asked on the site coordinator survey administered in the spring of 2012.

In terms of enrollment policies, site coordinators were asked to indicate the degree to which

activities provided at their site were:

Open to all students who want to participate

Only able to support limited enrollment and therefore filled on a first-come, first-served

basis

Based on giving enrollment priority to certain groups of students

Restricted in that only certain groups of students are eligible to participate

As Figure 14 shows, 51 percent of responding site coordinators indicated that all of the activities

provided at their site were open to all students who wanted to participate, and another 20 percent

indicated that most of their activities were open to all students. Clearly, the vast majority of the

centers active during the 2011–12 programming period provided activities that were largely open

to all students who wanted to participate. In contrast, only 10 percent of centers indicated that all

of the activities provided at their site were restricted in that only certain groups of students were

eligible to participate, and another 9 percent indicated that most of the activities they provided

were restricted.

Figure 14. Summary of Enrollment Policies Reported by Site Coordinators

51%

13% 16% 10%

20%

8% 16%

9% 15%

36%

23% 22% 15%

44% 45%

59%

0%

20%

40%

60%

80%

100%

Activities are open to allstudents that want to

participate

Activities are only able tosupport limited

enrollment and aretherefore filled on a firstcome, first served basis

Activities are based ongiving enrollment priority

to certain groups ofstudents

Activities are restricted inthat only certain groups

of students are eligible toparticipate

Re

spo

nd

ing

Site

Co

ord

inat

ors

All of the activities at this site

Most of the activities at this site

Some of the activities at this site

None of the activities at this site


In terms of recruitment approaches, site coordinators were asked a series of questions regarding

the extent to which students served at their site were recruited for enrollment in the program

based on the following:

The fact that the student scored “below proficient” on local or state assessments

The fact that the student failed to receive a passing grade during a preceding grading

period

A referral from school day staff because the student needed additional assistance in

reading or mathematics

The student’s status as an English language learner (ELL)

As Figure 15 displays, 43 percent of responding site coordinators indicated that most of the

student were enrolled in the program given that they had scored “below proficient” on local or

state assessments, and a majority of site coordinators indicated that some of the students had

been directed to the program because they failed to receive a passing grade during the preceding

grading period, were referred directly by school day staff, or were classified as an ELL student.

Figure 15. Summary of Enrollment Policies Reported by Site Coordinators

11%

43% 39%

7% 4%

25%

57%

14%

7%

32%

57%

5% 2%

11%

53%

34%

0%

20%

40%

60%

80%

100%

All of the students enrolledat this site

Most of the studentsenrolled at this site

Some of the studentsenrolled at this site

None of the studentsenrolled at this site

Re

spo

nd

ing

Site

Co

ord

inat

ors

The fact that the student scored “below proficient” on local or state assessments

The fact that the student failed to receive a passing grade during a preceding grading period

A referral from school day staff because the student needed additional assistance in reading ormathematics

The student’s status as an English Language Learner (ELL)


Summary of Grantee and Center Characteristics

Generally, the domain of Washington 21st CCLC grantees and centers operating during the

2011–12 reporting period were largely similar to grantees and centers nationwide in terms of

organizational and operational characteristics, although some differences were noted, as follows:

Washington grants were less likely to be held by school districts and more likely to

be held by regional/intermediate educational agencies (called educational service

districts in Washington) than in the nation as a whole.

Washington centers were less likely be staffed mostly by school day teachers.

Washington centers were more likely to adopt a mostly academic enrichment

program model when delivering activities.

It is not immediately clear if any significance should be attached to these differences between

Washington 21st CCLC grantees and the nation as a whole. Hypothetically, non-school-based

entities and programs less reliant on school day teachers may experience some additional

challenges in connecting activities to the school day or, at least at a minimum, may need to take

additional steps to ensure the necessary mechanisms to support communication and collaboration

are put in place. This theme will be explored more thoroughly in the leading indicator chapter

that follows as some of the leading indicators adopted for the 2011–12 programming period

pertain to the issue of linking 21st CCLC programming to the school day.


Chapter 4: Leading Indicators

Overview of Leading Indicators

A primary goal of the statewide evaluation was to provide 21st CCLC grantees with data to

inform program improvement efforts regarding their implementation of research-supported best

practices. AIR, the Weikart Center, and OSPI worked collaboratively to define a series of leading

indicators predicated on data collected as part of the statewide evaluation. The leading indicators

were meant to enhance existing information/data available to 21st CCLC grantees regarding how

they fare in the adoption of program strategies and approaches associated with high-quality

afterschool programming. Specifically, the leading indicator system was designed to do the

following:

Summarize data collected as part of the statewide evaluation in terms of how well the

grantee and its respective centers are adopting research-supported best practices.

Allow grantees to compare their level of performance on leading indicators with similar

programs and statewide averages.

Facilitate internal discussions about areas of program design and delivery that may

warrant additional attention from a program improvement perspective.

The leading indicator system is focused on quality program implementation as opposed to youth or

program outcomes. It is designed to feed existing data (from PPICS) and program evaluation data

back to programs regarding the adoption of research-supported practices, so programs can identify

strengths and weaknesses and reflect on areas of program design and delivery in need of further

growth and development. Figure 16 provides an overall depiction of the intention, purpose, and

process of the leading indicator system. More consistent implementation of research-supported

best practices will theoretically support the attainment of desired youth outcomes.

It is important to note that the indicators presented in this report are based on an initial attempt to

develop a leading indicator system for Washington 21st CCLC grantees, and it is anticipated that

the system will be refined and developed in future years. Although these measures are drawn

from the research literature, the evidence base linking performance on these particular measures

with the achievement of desired student outcomes is limited. In addition, many of the measures

are based on self-reported data and perceptions of program implementation provided by 21st

CCLC staff. As such, results should be treated with caution and not utilized to draw definitive

conclusions about the quality, approaches, and practices adopted by centers during 2011–12

operating period.


Figure 16. The Leading Indicator Process

Selected Leading Indicators

The nine adopted leading indicators are organized into four overarching contexts: (1) Organiza-

tional Context, focused on practices that occur among staff and management; (2) Instructional

Context, focused on practices that occur at the point of service, where staff and youth directly

interact; (3) Mutually Reinforcing Context, focused on practices related to coordinating and

aligning afterschool programming and activities with the regular school day, family, and

community contexts; and (4) Youth Outcome Leading Indicators, focused on the change in

students’ proficiency in reading/English language arts (ELA) and mathematics. Leading

indicators within each of these contexts are listed in Table 4.

Table 4. Leading Indicators by Context

1. Organizational Context

Leading Indicator 1.1 Staff Capacity

Leading Indicator 1.2 Continuous Improvement

Leading Indicator 1.3 Leadership and Management

2. Instructional Context

Leading Indicator 2.1 Quality of Instructional Content

Leading Indicator 2.2 Quality of Instructional Processes/Strategies

3. Mutually Reinforcing Context

Leading Indicator 3.1 Family Engagement

Leading Indicator 3.2 School Context

Leading Indicator 3.3 Community Context

4. Youth Outcome Leading Indicators

Leading Indicator 4.1 Reading/ELA and Mathematics Performance


Each of the adopted contexts and indicators are representative of AIR’s larger framework for

understanding the path to quality in afterschool programs. The achievement of desired youth

outcomes is a function of a complex set of interactions between several program elements:

Youth Characteristics. The characteristics and contributions youth bring to the afterschool

setting influence how they engage with and benefit from afterschool programs.

Community Context. The resources and characteristics of the local and school

community context serve to support meaningful partnerships to develop program goals,

program design, and provide program guidance.

Program Participation. Youth are more likely to benefit from afterschool program

participation if they attend consistently, over a period of time, and participate in a variety

of activity types.

Program Quality. Program quality is a series of practices and approaches that support

the provision of developmentally appropriate, high-quality settings and activities at the

point of service. This includes practices and approaches adopted by (a) activity leaders

working directly with youth (such practices are represented in the Instructional Context

domain in the leading indicator system) and (b) the organization as a whole, which

provides an infrastructure to support implementation of effective practice in the design,

delivery, and evaluation of afterschool programming (represented in the Organizational

Context and Mutually Reinforcing Context domains in the leading indicator system).

The current iteration of the leading indicator system addresses only a portion of the quality

framework depicted in Figure 17; there are a number of opportunities to expand the leading

indicator system to more fully represent additional, important components of afterschool

program quality.

Figure 17. AIR’s Quality Framework for Afterschool Programs


Organization of Leading Indicators Chapter

This chapter is organized, first and foremost, by the four broad contexts included in the leading

indicators. Within each context, data associated with a leading indicator in a given context are

summarized (for Washington centers overall). Two primary approaches to summarizing state-

level leading indicator data were used, as follows:

Scaled Items. Many questions on the site coordinator and staff surveys are part of a

series of questions designed to assess an underlying construct/concept and result in a

single scale score summarizing performance on a given aspect of a leading indicator (e.g.,

practices that support linkages to the school day). For these scale scores, Rasch scale

scores were created using staff and site coordinator responses to a series of survey

questions to create one overall score. Indicators analyzed using Rasch scales include a

scale score ranging from 0 to 100, where higher scores are indicative of a higher level or

more frequent adoption of a leading indicator. Average scale scores and the distribution

of scale scores across the response categories for a given scale are provided. For example,

a mean value of 56.57 may put the statewide average for a given indicator in the agree

range of the scale with response options for strongly disagree, disagree, agree, and

strongly agree). Site coordinator scale scores represent responses from one site

coordinator, and center scale scores represent the average of scale scores for all staff

respondents associated with a given center.

Descriptive Items. Other leading indicators are based on data that are not appropriate for

the type of scale construction just described. For example, program objectives are stand-

alone items that do not necessarily contribute to an underlying construct or concept. Items

of this type are summarized descriptively.


Leading indicators within the Organizational Context examine both staff development and

internal communication and collaboration among program staff. As noted by Smith (2007),

Glisson (2007), and Birmingham, Pechman, Russell, and Mielke (2005), an organizational

climate that supports staff in reflecting on and continually improving program quality is a key

aspect of effective youth development programs. Programs characterized by a supportive and

collaborative climate permit staff to engage in self-reflective practice to improve overall program

quality. Self-reflective practice is more likely to lead to high-quality program sessions that

provide youth with positive and meaningful experiences. Three leading indicators fall under the

Organizational Context: (1) Staff Capacity; (2) Continuous Improvement, which is assessed by

scales measuring program climate and internal communication and collaboration; and (3)

Leadership and Management.

Leading Indicator 1.1: Staff Capacity

This leading indicator is meant to capture the degree to which staff receive training on delivering

high-quality instruction and are supported by middle and upper management in their efforts to

participate in professional development and training. Staff were asked a series of survey

questions related to the types of training they participated in during the 2011–12 school year.

As shown in Table 5, 63 percent of respondents reported participating in some type of training


related to their role in the afterschool program. It should be noted that there was a difference in

terms of the most frequent types of training reported by staff between the 2010–11 and 2011–12

school years. As shown in Table 6, while the training for providing effective academic

enrichment activities remained as one of the most frequent types, the other one shifted from

providing activities to support youth development to maintaining healthy and safe environment.

Although data were not available to explore if participation in the YPQI process influenced these

results, it is assumed that participation in the YPQI initiative had some level of impact on staff

responses.

Table 5. Staff Responses to Questions about Training Participation

PROMPT: Which of the following types of training were required and/or offered to you

during the present school year, and which did you attend or do you plan to attend in the

future? Please check all that apply.

Statewide

(N = 989)

% of Responders Reporting Participation in 21st CCLC Training 63%

Delivering effective enrichment activities. 61%

Providing activities to support youth development. 58%

Maintaining healthy and safe environments. 61%

Providing academic content in an afterschool setting. 46%

Learning how to apply principles related to child and adolescent growth and

development to activity design and delivery. 41%

Conflict resolution and behavior management. 50%

Working with a diverse student population. 49%

Parent and family engagement. 42%

Using data on student needs to inform programming. 34%

Providing activities that support college and workforce readiness. 18%

Other 5%

Data Source: Staff Survey

Table 6. The Most Frequent Types of Training Reported by Staff

Type of Training 2010-2011 School Year 2011-2012 School Year

Providing effective academic

enrichment activities √ √

Providing activities to support youth

development √

Maintaining healthy and safe

environment √



Leading Indicator 1.2: Continuous Improvement

Data for this leading indicator are summarized with Rasch scale scores ranging from 0 to 100,

where higher scores are indicative of higher levels of performance or endorsement of a given

scale. Three Rasch scale scores were calculated for this indicator to summarize the following

aspects of continuous improvement:

Program Climate: The extent to which program staff report that a supportive and

collaborative climate exists within the program (from the staff survey)

Internal Communication—Site Coordinator: How frequently site coordinators engage

in practices that support internal staff communication and collaboration (from the site

coordinator survey)

Internal Communication—Staff: How frequently staff engage in internal

communication and collaboration (from the staff survey)

Program Climate

Scale scores for program climate are based on the following question from the staff survey:

PROMPT: Please rate the extent to which you agree or disagree with the following with respect

to climate in your program:

There is adequate time to focus on individual student needs within the program time

frame.

The program staff has shared control over the content.

The staff is encouraged to try new and innovative approaches.

Instructional collaboration among program staff is encouraged and supported.

Staff is provided with training in current research on best practices in afterschool

programs.

Staff participate fully in program decision making.

There is adequate time to plan individual activity sessions.

As shown in Table 7, the statewide average scale score on the program climate fell within the

agree range of the scale (scale response options included strongly disagree, disagree, agree, and

strongly agree), suggesting that most staff reported supportive, collaborative program climates.

This conclusion is echoed in Figure 18, summarizing the distribution of centers in the four

response categories. A majority of centers (75.56 percent) fell in the agree response category,

and 21.67 percent fell in the disagree or strongly disagree response category. It should be noted

that there was a difference regarding the two statements staff were most likely to disagree with

between the 2010–11 and 2011–12 school years. As shown in Table 8, although There is

adequate time to plan individual activity sessions remained as one of the two statements for both

programming periods, the other one shifted from Staff is provided with training in current

research on best practices in afterschool programs to Staff participate fully in program decision

making. In these instances, there are ways OSPI can better support afterschool staff. For

example, future requests for proposal (RFPs) can be modified to require that programs build in

time for session planning or offer and support staff participation in program decision making.


Table 7. Statewide Performance on the Program Climate Scale

Statewide Mean

Response Category

for Statewide Mean

Program Climate The extent to which program staff report that a supportive

and collaborative climate exists within the program.

59.71

(N = 1053)

Agree

(N = 1053)


Table 8. Statements That Staff Are Most Likely To Disagree With

Question 2010–11 School Year 2011–12 School Year

There is adequate time to plan

individual activity sessions √ √

Staff is provided with training in

current research on best practices in

afterschool programs

√

Staff participated fully in program

decision-making √



Figure 18. Distribution of Centers in Response Categories for Survey Questions About

Program Climate

Source: Staff Survey (1,053 responses from 180 centers)

Internal Communication

Scale scores of internal communication included staff and site coordinator responses to the

following survey question:

PROMPT: How often do you engage in the following tasks with other staff working in the program?

Conduct program planning based on a review of program data with other staff.

Use data to set program improvement goals with other staff.

Discuss progress on meeting program improvement goals with other staff.

Observe other afterschool staff delivering programming in order to provide feedback on

their practice.

Conduct program planning with other staff in order to meet specific learning goals in

coordinated ways across multiple activities.

Share ideas with other staff on how to make programming more engaging for

participating students.

Share experiences and follow up about individual youth and other staff.

Engage in discussions with other staff and school day teachers and/or administrators on

how the program could better support student learning needs.

1.67%

20.00%

75.56%

2.78%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Strongly Disagree Disagree Agree Strongly Agree

Cen

ters


Participate in training and professional development with other staff on how to better

serve youth.

Discuss current research-based instructional practices with other staff.

The average statewide scale score for internal communication fell within the a couple of times

per year response category for site coordinators (scale response options included never, a couple

of times per year, about once a month, and nearly every week), suggesting the assessed

collaborative efforts were relatively infrequently implemented during the 2011–12 programming

period. For staff, the statewide average scale score also fell within the a couple of times per year

response category (see Table 9).

As shown in Figures 19 and 20, 64.71 percent of site coordinators fell in the a couple of times

per year response category, and 26.52 percent of centers fell within the same response category.

Staff survey respondents fell in the about once a month response category, with 59.67 percent of

centers falling in this response category (see Figure 20). These results may suggest that staff

members are slightly more likely to engage with one another in types of internal communication

assessed by the scale as opposed to engaging in internal collaboration with their site

coordinators. For staff, the least frequently implemented internal communication activity was to

Use data to set program improvement goals with other staff, although it was Observe other

afterschool staff delivering programming in order to provide feedback on their practice in 2010–

11 programming period (see Table 10). Yet, we anticipate these results may vary across sites

enrolled in the YPQI initiative versus those that were not.

Table 9. Statewide Performance on the Internal Communication Scale

Statewide Mean

Range of Scale

Statewide Mean

Fell Within

Internal Communication—Site Coordinator Survey The frequency with which the site coordinator engages in

practices with program staff that support internal

communication and collaboration.

59.26

(N = 185)

A couple of times

per year

(N = 185)

Internal Communication—Staff Survey The frequency with which the staff engages in practices

with other program staff that support internal

communication and collaboration.

59.75

(N = 1,026)

A couple of times

per year

(N = 1,026)

Data Source: Site Coordinator and Staff Surveys


Figure 19. Distribution of Site Coordinators in Response Categories for Survey Questions

About Internal Communication

Source: Site Coordinator Survey (185 responses)


Internal Communication Based on Staff Survey Responses


2.35%

64.71%

29.41%

3.53%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

Never A Couple of Times

Per Year

About Once a Month Nearly Every Week

Sit

e C

oo

rdin

ato

rs

3.31%

26.52%

59.67%

10.50%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

Never A Couple of Times

Per Year

About Once a Month Nearly Every Week

Cen

ters


Table 10. The Least Frequently Implemented Internal Communication Activity

Internal Communication Activity 2010–11 School Year 2011–12 School Year

Use data to set program improvement

goals with other staff. √

Observe other afterschool staff

delivering programming in order to

provide feedback on their practice.

√


Leading Indicator 1.3: Leadership and Management

This leading indicator is meant to capture the degree to which the program has taken steps to hire

qualified staff, promote staff development, support program improvement, and solicit feedback.

Some of these areas overlap with previously identified indicators in the Organizational Context

domain, but the data presented in relation to this indicator directly represent how the program

believes it is doing in carrying out leadership and management tasks. This indicator is based on

data obtained from the Form B of Youth Program Quality Assessment (YPQA), a validated

instrument designed to evaluate the quality of youth programs and identify staff training needs.

The YPQA Form B focuses on program quality at the organizational level and assesses the

quality of organizational supports for the youth program offering assessed in Form A.

Staff were asked a series of questions regarding staff availability and longevity with the center,

qualifications, staff development, and ongoing program improvement. As shown in Table 11, the

statewide average scale score for leadership and management fell within the three range of the

scale (scale response options included one, three, and five), suggesting that most staff reported

the leadership and management in the center support youth-staff relationships and a positive

development focus, promote staff development, and are committed to ongoing program

improvement. This conclusion is echoed in Figure 21, indicating a majority of centers (77.42

percent) fell in the three response category, and 9.68 percent fell in the five response category.

Table 11. Statewide Performance on the Leadership and Management Scale

Statewide Mean

Response Category

for Statewide Mean

Leadership and Management – YPQA Form B

The extent to which the program is engaging in

practices that ensure staff are well positioned to

create developmentally appropriate settings for youth

and that processes are in place to support program

improvement efforts

52.43

(N = 62)

Three

(N = 62)

Data Source: YPQA Form B



Leadership and Management

Source: YPQA Form B

Summary of Organizational Context Findings and Recommendations

As previously noted, the leading indicator system is part of a larger infrastructure constructed by

OSPI to support 21st CCLC-funded program improvement. This larger infrastructure includes the

YPQI quality improvement process. During the course of the 2011–12 programming period,

roughly half of active centers participated in the YPQI initiative on a voluntary basis. Although not

formally examined, it is hypothesized that YPQI sites were more apt to report engaging in the

types of practices and approaches described in the Organizational Context leading indicators.

In light of this, it is recommended that OSPI consider mandating participation in a YPQI-like

process for 21st CCLC grantees during some point of their five-year grant period. From a policy

perspective, OSPI also may want to consider making modifications to future 21st CCLC RFPs to

articulate this requirement and include standard budget line items where sites can identify the

resources they will dedicate to staff participation in YPQI-like processes. This will help ensure

that the value and importance of quality monitoring activities are relayed to programs and that

programs secure the necessary resources to effectively participate in such efforts.


Leading indicators in the Instructional Context focus on the practices and approaches adopted by

frontline staff to design and deliver activity sessions that intentionally support youth skill

building and mastery that align with the center’s objectives and principals of youth development.

There is a strong connection between the leading indicators in the Instructional Context and

components of the YPQI program improvement process. For example, the YPQI process

12.90%

77.42%

9.68%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

One Three Five

Cen

ters


assesses and supports staff practices at the point of service related to creating safe, supportive,

interactive, and engaging environments. The benefits of intentional design are also reflected in

the work of Durlak and Weissberg (2007), who found that effective afterschool programs

commonly provided activities that were sequenced, involved active forms of learning, and

focused on cultivating particular skills. There are two leading indicators in the Instructional

Context: (1) Quality of Instructional Content and (2) Quality of Instructional Processes/

Strategies.

Leading Indicator 2.1: Quality of Instructional Content

This leading indicator is meant to capture the degree to which the time spent on activities

corresponds to program objectives as identified by site coordinators and how intentionally

activities are designed and delivered. Both descriptive and Rasch scaling approaches were used in

relation to these data. Three separate metrics were calculated to describe aspects of this indicator:

Program Objectives: The degree to which site coordinators’ top three objectives align

with the proportion of time spent on corresponding activities

Intentionality in Program Design—Site Coordinator Survey: The frequency with which

staff engages in practices that indicate intentionality in activity and session design for the

delivery of activities meant to support student growth and development in reading and

mathematics

Intentionality in Program Design—Staff Survey: The frequency with which staff

engages in practices that indicate intentionality in activity and session design for the

delivery of activities meant to support student growth and development

Program Objectives and Alignment With Time Dedicated to Corresponding Activities

In order to assess alignment between activity provision and the program objectives identified by

the center in question, site coordinators were asked to rank their top three program objectives. In

order to assess alignment, steps were then taken to define each objective in regard to the proportion

of total activity time that could be minimally dedicated to particular activities to meet the identified

program objective. For example, if a site coordinator indicated that a primary program objective

was to enable low-performing students to achieve grade-level proficiency, then it was expected that

a certain number of hours would be dedicated to providing activities designed to support skill

building in core academic areas. As shown in Table 12, on average, 90 percent of center objectives

aligned with the actual frequency of activities provided during the programming period.

As shown in Table 12, the most common top three program objectives included (1) raise the

academic performance levels of any students who have an interest in participating (79 percent

endorsing), (2) provide students with access to academic enrichment opportunities (64 percent

endorsing), and (3) enable low-performing students to achieve grade-level proficiency (43 percent

endorsing). Based on these results, it is clear that most 21st CCLC site coordinators understand the

importance of prioritizing student academic growth and development and providing activities that

align with this priority.


Table 12. Statewide Performance on the Program Objectives Aligned to Activity Provision

PROMPT: Please indicate which of these program objectives

constitute the top three priorities for your program. Statewide

(N = 188)

Program Objectives—the degree to which site coordinators’ top

three objectives align with proportion of time spent on

corresponding activities.

90%

Percentage of Site Coordinators Indicating Objective Was

Among Their Top Three Priorities

Enable low-performing students to achieve grade-level

proficiency. 43%

Provide students with access to academic enrichment

opportunities. 64%

Raise the academic performance levels of any students who

have an interest in participating. 79%

Provide supervised space for students to complete homework. 21%

Enhance the social or civic development of students. 29%

Prepare students for college and work. 29%

Provide students with the opportunity to participate in sports and

recreation activities. 21%

Enhance the artistic development of students (e.g., visual and

performing arts, etc.). 7%

Other 7%

Data Source: Site Coordinator Survey and PPICS

Intentionality in Program Design

As previously noted, a growing body of research suggests that program outcomes in the form of

enhanced student academic achievement outcomes are realized by simply paying attention to

how programming is delivered—specifically, whether or not programming is delivered in

developmentally appropriate settings grounded in core principles of youth development

(Birmingham et al., 2005; Durlak & Weissberg, 2007). In addition to youth development

principles, afterschool programs are more likely to attain desired student academic outcomes if

staff members responsible for planning the content of sessions incorporate certain practices and

strategies into their planning efforts.

On both the site coordinator and staff surveys, a series of questions was asked about intentional

program design.

Scale scores for intentionality in program design included staff and site coordinator responses to

the following survey questions:


PROMPT: How often do staff lead activities that are especially meant to support student growth

and development in reading and/or mathematics and provide program activities that are ...

Based on written plans for the session, assignments, and projects?

Well planned in advance?

Tied to specific learning goals?

Meant to build upon skills cultivated in a prior activity or session?

Explicitly meant to promote skill building and mastery in relation to one or more state

standards?

Explicitly meant to address a specific developmental domain (e.g., cognitive, social,

emotional, civic, physical, etc.)?

Informed by the express interests, preferences, and/or satisfaction of participating youth?

Although the items appearing on each survey were the same, site coordinators were asked to

indicate how frequently staff leading activities to support skill building in reading and/or

mathematics engaged in the practices listed above, and staff were asked how frequently they

engaged in these practices. It should be noted that some differences between site coordinator and

staff responses to the above survey questions may be associated with the fact that staff who are

not responsible for leading activities that support skill building and mastery in reading and

mathematics also completed surveys and were included in the analysis.

As shown in Table 13, the average site coordinator scale scores fell within the frequently

response category (response options were rarely, sometimes, frequently, and always), suggesting

that site coordinators felt practices related to intentional service delivery are commonly adopted.

Average staff scale scores also fell in the frequently response category. As Figures 22 and 23

show, 44.71 percent of site coordinator scale scores fell in the frequently response category, and

73.53 percent of centers fell in the frequently response category. This indicates that staff is more

likely to report engaging in practices related to intentional program design relative to site

coordinator responses of how frequently staff engage in practices related to intentional program

design.

It is possible that differences between site coordinator and staff responses suggest that some staff

are acting in a more autonomous fashion when planning activities, operating outside of any

organizational structures or criteria for planning activity sessions. Generally, this is an area that

warrants additional attention by OSPI, particularly in reference to the previously discussed

program climate findings that a substantial proportion of frontline staff struggle to find adequate

time to plan activity sessions and offerings.


Table 13. Statewide Performance on the Intentionality in Program Design Scale

Statewide Mean

Range of Scale

Statewide Mean

Fell Within

Intentionality in Program Design—Site Coordinator

Survey The frequency with which staff engage in practices that

indicate intentionality in activity and session design among

staff responsible for the delivery of activities meant to

support student growth and development.

59.82

(N = 185)

Frequently

(N = 185)

Intentionality in Program Design—Staff Survey The frequency with which staff engage in practices that

indicate intentionality in activity and session design among

staff responsible for the delivery of activities meant to

support student growth and development.

59.64

(N = 1,051)

Frequently

(N = 1,051)

Data Source: Site Coordinator Survey

Figure 22. Distribution of Site Coordinator Scale Scores From Survey Questions About

Intentional Program Design


8.24%

36.47%

44.71%

10.59%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

Rarely Sometimes Frequently Always

Sit

e C

oo

rdin

ato

rs


Figure 23. Distribution of Center Scale Scores From Survey Questions About Intentional

Program Design Based on Staff Survey Responses


Leading Indicator 2.2: Quality of Instructional Processes/Strategies

This leading indicator is meant to capture the processes and practices in which staff members

engage that are consistent with high-quality instruction and core youth development principles,

with particular emphasis on providing developmentally appropriate activities at the point of

service. Conceptually, many of the practices associated with this indicator are related to the

concepts embedded in YPQA. All of the data reported in relation to this indicator were scored

using Rasch scale scores ranging from 0 to 100, where higher scores are indicative of higher

performance/higher frequency on the assessed aspects of leading indicator 2.2. Six separate scale

scores were calculated to assess aspects of this leading indicator:

Point of Service Quality—YPQA Form A: The extent to which program staff provide

supports and opportunities to create safe, supportive, interactive, and engaging settings

for participating youth

Youth-Centered Policies and Practices—YPQA Form B: The extent to which the

program adopts youth-centered policies and practices conducive to a supportive learning

environment

Youth Ownership—Site Coordinator Survey: The extent to which the site coordinators

perceive that program staff provide opportunities to develop youth ownership in the

program

Youth Ownership—Staff Survey: The extent to which program staff perceive the

presence of opportunities to develop youth ownership

2.35%

22.94%

73.53%

7.06%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

Rarely Sometimes Frequently Always

Cen

ters


Capacity to Create Interactive and Engaging Settings: The extent to which staff

members perceive that programming provides interactive and engaging settings for

participating youth

Service Delivery Practices: The frequency with which staff adopt specific practices that

support youth development

Point of Service Quality

This leading indicator is assessed by scales measuring safety, supportive environment,

interaction, and engagement. Table 14 displays both self-assessment and/or external assessment

data obtained by scoring the YPQA Form A observational tool. Scores were placed on a 0 to 100

scale and were adjusted to account for the bias introduced by the type of assessor (i.e., external

or self-assessment) and the type of activity observed (i.e., enrichment, tutoring/homework help;

or recreation).

As shown in Table 14, the average scale scores for the safe environment and the supportive

environment fell within the functioning near optimal category, and those for the interaction and

engagement fell within the still room for improvement category. But the average statewide scale

score for overall point of service quality fell within the functioning near optimal category.

This conclusion is echoed in Figures 24 through 28. For point of service quality, safe

environment and supportive environment, the percentages of staff respondents that fell within the

functioning near optimal category are 80.56 percent, 100 percent, and 88.89 percent,

respectively. In contrast, the percentages of staff respondents that fell within the still room for

improvement category for interaction and engagement are 83.33 percent and 90.28 percent.


Table 14. Statewide Performance on the Point of Service Quality Scale

Statewide Mean

Response Category

for Statewide Mean

Point of Service Quality – YPQA Form A

The extent to which program staff provide supports and

opportunities to create safe, supportive, interactive, and

engaging settings for participating youth (total YPQA

score).

53.55

(N = 219)

Functioning Near

Optimal

(N = 219)

Safe Environment 68.48

(N = 219)

Functioning Near

Optimal

(N = 219)

Supportive Environment 57.35

(N = 219)

Functioning Near

Optimal

(N = 219)

Interaction 39.25

(N = 219)

Still Room for

Improvement

(N = 219)

Engagement 33.06

(N = 219)

Still Room for

Improvement

(N = 219)

Data Source: Form B PQA


Point of Service Quality

Source: YPQA Form B

80.56%

19.44%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

Functioning Near Optimal Still Room for Improvement

Cen

ters



Safe Environment

Source: YPQA Form B


Supportive Environment

Source: YPQA Form B

100.00%

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

Functioning Near Optimal

Cen

ters

88.89%

11.11%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%


Cen

ters



Interaction

Source: YPQA Form B


Engagement

Source: YPQA Form B

16.67%

83.33%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%


Cen

ters

9.72%

90.28%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

Functioning Near Optimal Still Room for

Improvement

Cen

ters


Youth-Centered Policies and Practices

This leading indicator is meant to capture the degree to which the program adopts youth-centered

policies and practices conducive to a supportive learning environment. The data presented in

relation to this indicator are based on data obtained from the YPQA Form B. Staff were asked a

series of questions about the program’s relevance to youth interests and skills, as well as youths’

influence on the setting, activities, structure, and policy of the center. As shown in Table 15, the

statewide average scale score for youth-centered policies and practices fell within the three

range of the scale (scale response options included one, three, and five). Figure 29 shows that a

majority of centers (75.81 percent) fell within the three category, while 9.68 percent fell in one

category. This indicates that most staff reported programs tap youth interests, build multiple

skills, and involve youth in the settings, activities, structure and policy of the program.

Table 15. Statewide Performance on the Youth-Centered Policies and Practices Scale

Statewide Mean

Response Category

for Statewide Mean

Youth-Centered Policies and Practices – YPQA Form B

The extent to which the program adopts youth-centered

policies and practices and practices conducive to a

supportive learning environment.

52.42

(N = 62)

Three

(N = 62)




Youth-Centered Policies and Practices

Source: YPQA Form B

Youth Ownership

Youth ownership refers to allowing youth to shape the afterschool program by setting goals for

what they want to accomplish in the program, making choices about both the content and process

of delivering offerings, planning activities, and having a role in governing the program.

Scale scores for youth ownership include staff and site coordinator responses to the following

survey question:

PROMPT: Please indicate your level of agreement with the following statements about how your

students build ownership of the program:

Youth are afforded opportunities to take responsibility for their own program.

Youth have the opportunity to set goals for what they want to accomplish in the program.

Youth help make plans for what activities are offered at the program.

Youth make choices about what content is covered in program offerings.

Youth make choices about how content is covered in program offerings.

Youth help create rules and guidelines for the program.

9.68%

75.81%

14.52%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

One Three Five

Cen

ters


As shown in Table 16, the average site coordinator scale scores fell in the disagree category, and

staff survey scale scores fell in the agree response category (response options were strongly

disagree, disagree, agree, and strongly agree), suggesting that, on average, centers were

incorporating youth-ownership-related practices into their work with youth. Figures 30 and 31,

however, show that both the majority of site coordinator responses (51.18 percent) and of center

responses (59.67 percent) fell within the agree category. It should be noted that there was a

significant difference from the result in the evaluation report for the 2010–11 school year, where

54 percent of site coordinator respondents fell in the disagree category and 50 percent of center

responses fell in the agree category. This change may indicate that site coordinators and staff,

although with disparate perceptions previously, tend to agree on staff members’ efforts to

develop youth ownership in the program.

Table 16. Statewide Performance on the Youth Ownership Scale

Statewide Mean

Range of Scale

Statewide Mean

Fell Within

Youth Ownership—Site Coordinator Survey The extent to which the site coordinator perceives program

staff extending opportunities to youth to develop ownership

in the program.

59.19

(N = 185)

Disagree

(N = 185)

Youth Ownership—Staff Survey The extent to which program staff perceive staff members

extending opportunities to youth to develop ownership in

the program.

59.93

(N = 1030)

Agree

(N = 1030)



Figure 30. Distribution of Site Coordinator Scale Scores for Survey Questions About

Youth Ownership


Figure 31. Distribution of Center Scale Scores for Survey Questions About Youth

Ownership Based on Staff Survey Responses


1.18%

46.47%

51.18%

1.18%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%


Per

cen

t

1.11%

37.02%

59.67%

2.21%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%


Cen

ters


Capacity to Create Interactive and Engaging Settings

Unlike previously described scales, questions related to the capacity to create interactive and

engaging settings ask staff to rate the collective practices of all frontline staff as opposed to

rating their own practice. There is an assumption that staff working in the program have

sufficient interactions with other frontline staff to provide accurate ratings of how well the center

as a whole implements practices that will result in an interactive and engaging setting.

Scale scores for the capacity to create interactive and engaging settings include staff responses to

the following survey question:

PROMPT: Please rate the extent to which you agree or disagree with the following statements

regarding all staff that work with students in this program:

Program staff provides youth the opportunity to engage in group discussion and dialogue

more than placing youth in the role of passive listeners to a lesson or lecture delivered by

staff.

Program staff actively and continuously consults and involves youth.

Program staff provides structured and planned activities explicitly designed to help youth

get to know one another.

Program staff provides opportunities for youth to lead activities.

Program staff provides opportunities for youth to help or mentor other youth in

completing a project or task.

Program staff provides opportunities for the work, achievements, or accomplishments of

youth to be publicly recognized.

Program staff provides ongoing opportunities for youth to reflect on their experiences

(e.g., formal journal writing, informal conversational feedback).

Program staff is effective at finding ways to provide youth with meaningful choices when

delivering activities.

Program staff is effective at providing youth with opportunities to set goals and make

plans within the confines of the program.

Program staff asks for and listens to student opinions about the way things should work

in the program.

As shown in Table 17, the statewide average scale score on the capacity to create interactive and

engaging settings scale fell within the agree response category, suggesting that staff generally

perceive that frontline staff adopt many of the practices listed above. As shown in Figure 32, the

majority (69.41 percent) of centers fell in the agree response category, and 12.35 percent fell in

the disagree response category. This is consistent with prior findings from the evaluation team,

where staff are more confident in their collective ability to create interactive and engaging

settings for youth as compared with their individual practice.


Table 17. Statewide Performance on the Capacity to Create Interactive and Engaging

Settings Scale

Statewide Mean

Range of Scale

Statewide Mean

Fell Within

Capacity to Create Interactive and Engaging Settings The extent to which staff members perceive program staff

extending opportunities and providing supports to youth

that result in the creation of an interactive and engaging

setting for participating youth.

60.95

(N = 185)

Agree

(N = 185)


Figure 32. Distribution of Center Scale Scores for Survey Questions About

Staff Capacity to Create Interactive and Engaging Settings Based on Staff Survey

Responses

Source: Staff Survey (185 responses from 180 centers)

Service Delivery Practices

Finally, the service delivery practices scale focuses on how frequently staff report adopting

practices that are likely to foster an interactive and engaging environment for participating youth

(i.e., individual as opposed to collective practice). The list of practices represented on the scale is

in no way meant to be comprehensive but is generally aligned to specific youth development

principles represented on the YPQA.

12.35%

69.41%

18.24%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

Disagree Agree Strongly Agree

Cen

ters


Scale scores for service delivery practices include staff responses to the following survey question:

PROMPT: How often are students who are participating in the activities you provide in the

program afforded the following types of opportunities?

Work collaboratively with other students in small groups.

Have the freedom to choose what activities or projects they are going to work on or

participate in.

Work on group projects that take more than one day to complete.

Lead group activities.

Provide feedback on the activities they are participating in during time set aside explicitly

for this purpose.

Participate in activities that are specifically designed to help students get to know one

another.

Make formal presentations to the larger group of students.

As shown in Table 18, the average scale score for staff on the service delivery practices scale fell

within the available occasionally response category (response options included never available,

available occasionally, available regularly, and always available). As Figure 33 shows, 56.35

percent of centers fell in the available occasionally response category.

Compared with staff responses for the 2010–11 school year, the least frequently provided

opportunities shifted from Lead group activities and Make formal presentations to the larger

group of students to Participate in a sequence of sessions where task complexity increases to

build specific skills and Make formal presentations to the larger group of students (see Table

19). Here again, there are certainly opportunities for growth, and it would seem that the YPQI

initiative is conducive to supporting further adoption of these practices in developmentally

appropriate ways.

Table 18. Statewide Performance on the Service Delivery Practices Scale

Statewide Mean

Range of Scale

Statewide Mean

Fell Within

Service Delivery Practices—The frequency with which

staff adopt practices that support youth development.

58.7

(N = 1,066)

Available

Occasionally

(N = 1,066)



Table 19. The Least Frequently Provided Opportunities

Opportunity 2010-2011 School Year 2011-2012 School Year

Make formal presentations to the larger

group of students √ √

Lead group activities √

Participate in a sequence of sessions

where task complexity increases to

build specific skills

√


Figure 33. Distribution of Center Scale Scores for Survey Questions About

Service Delivery Practices Based on Staff Survey Responses


Summary of Instructional Context Findings and Recommendations

In many respects, of all the leading indicators, those within the instructional context are

potentially of greatest importance in ensuring high-quality programming due to the fact that

quality at the point of service is how youth experience and benefit from programming. On

average, centers are doing reasonably well in adopting both content- and process-related

practices and approaches associated with the (1) alignment of program activities to identified

program objectives and (2) intentionality of content delivered and support of the provision of

developmentally appropriate settings. However, there is room for growth in each of these areas,

particularly in relation to enhancing intentionality in activity session design and delivery and

providing opportunities for youth ownership.

0.55%

56.35%

38.67%

4.42%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Never Available Available

Occasionally

Available Regularly Always Available

Cen

ters


In order to better target program improvement efforts, more information is needed about the

following areas:

How does performance on leading indicators in the instructional context vary by the

grade level of students served by the program? The concern here is that several of the

process indicators may be more relevant to programs serving youth in secondary grades,

which may warrant exploring different measurement strategies depending on the grade

level of the youth in question.

How does performance on leading indicators in the instructional context vary by center

enrollment in the YPQI process? Here again, many of the items underpinning the

indicators are conceptually related to YPQA, and it is anticipated that centers enrolled in

YPQI will have higher scale scores as a result. Verification of this hypothesis would

enhance the validity of the data resulting in the survey process.

How much variation exists within and across centers in terms of the adoption of high-

quality instructional practice, and how can this information be communicated to centers

in a way to support program improvement efforts without penalizing individual centers

and/or staff?

Each of these questions has implications for the continued development of the leading indicator

system and warrants further exploration in Year 3 evaluation efforts.


The Mutually Reinforcing Context focuses on relationships between the 21st CCLC program

and contexts external to the program that significantly impact the success of the program.

Community partners, families, and schools play an important role in 21st CCLC programs by

expanding program activities, facilitating program sustainability, and providing important

information about student needs. Three leading indicators are associated with the Mutually

Reinforcing Context: (1) Family Engagement, (2) School Context, and (3) Community Context.

Indicator 3.1: Family Engagement

Engaging families in programming and providing family learning events is an important

component of 21st CCLC programs. Programs may engage families by communicating with

them about center programming and events, collaborating to enhance their child’s educational

success, and providing family literacy and/or social events. Survey questions on the site

coordinator survey measured center approaches to family communication.

Scale scores for family engagement included site coordinator responses to the following survey

questions:

PROMPT: How often do you ...

Send materials about program offerings home to parents/adult family members?

Send information home about how the student is progressing in the program?

Hold events or meetings to which parents/adult family members are invited?


Have conversations with parents/adult family members over the phone?

Meet with one or more parents/adult family members?

Ask for input from parents/adult family members on what and how activities should be

provided?

Encourage parents/adult family members to participate in center-provided programming

meant to support their acquisition of knowledge or skills?

Encourage parents/adult family members to participate in center-provided programming

with their children?

As shown in Table 20, the average family communication scale score fell within the sometimes

response category (response options were never, sometimes, and frequently), which is indicative

of programs typically communicating with families once or twice a semester. As Figure 34

shows, 73.53 percent of site coordinator respondents fell in the sometimes response category.

The least common family communication strategies included sending information home about

how the student is progressing in the program and asking for input from parents/adult family

members on what and how activities should be provided. The former is not surprising given the

difficulty associated with providing individual progress reports on specific students. However,

the latter is more surprising considering that obtaining feedback from parents/adult family

members is not an overly burdensome or costly task. There might be an opportunity for local

evaluators to assist programs in collecting feedback from parents/adult family members.

Table 20. Statewide Performance on the Family Communication Scale

Statewide Mean

Range of Scale

Statewide Mean

Fell Within

Family Communication The frequency with which staff adopt practices that support

communication with parents and adult family members.

59.64

(N = 185)

Sometimes

(N = 185)



Figure 34. Summary of Site Coordinator Responses Regarding Family Engagement


Indicator 3.2: School Context

This leading indicator is meant to capture the degree to which 21st CCLC staff members align

the design and delivery of programming to the school day and individual student needs. These

practices are particularly important to 21st CCLC program quality, given the explicit goal of

supporting low-performing students’ growth in reading and mathematics. The data reported for

this leading indicator were scored with Rasch scale scores ranging from 0 to 100, where higher

scores are indicative of higher performance/endorsement on a given scale. Four separate scale

scores were calculated for this indicator:

Linkages to the School Day—Site Coordinator Survey: The extent to which the site

coordinator reports taking steps to establish links to the school day and use student data

to inform programming

Linkages to the School Day—Staff Survey: The extent to which program staff report

taking steps to establish links to the school day and use student data to inform

programming

Data Use—Site Coordinator Survey: The extent to which the site coordinator reports

the program using student data to inform programming

Data Use —Staff Survey: The extent to which program staff report taking steps to use

student data to inform programming

It is important to note that the items for linkages to the school day scales on the site coordinator

and staff surveys were quite different. On the site coordinator survey, items were designed to ask

about specific strategies adopted by the program to establish meaningful links to the school day.

11.76%

73.53%

14.71%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

Never Sometimes Frequently

Sit

e C

oo

rdin

ato

rs


Site coordinators were asked to indicate if the strategy described in a given item was (1) a major

strategy, (2) a minor strategy, or (3) not a strategy to support links with the school day. In

contrast, the staff survey asked respondents to indicate their level of agreement with a series of

items regarding their knowledge of school day practices, student academic needs, use of student

data to inform programming, and communication with school day staff to better support the

design and delivery of afterschool programming.

Scale scores for site coordinator responses included the following survey question:

PROMPT: Please indicate whether you receive each of the following, and to what extent you use

it in planning for the activities you provide:

The program has access to individualized education plans, and staff use this information

to plan activities.

The program has access to students’ state assessment scores, and staff use this

information to plan activities.

The program has access to students’ scores on district- or building-level assessments, and

staff use this information to plan activities.

The program has access to students’ grades, and staff use this information to plan

activities.

The program has access to teacher-provided student progress reports/teacher input, and

staff use this information to plan activities.

Scale scores for staff survey responses included the following survey questions:

PROMPT: Please rate the extent to which you agree or disagree with the following statements

regarding linkages to the school day:

On a week-to-week basis, staff know what academic content will be covered during the

school day with the students they work with in the afterschool program.

Staff coordinate the content of the afterschool activities they provide with my students’

school day homework.

Staff know who to contact at their students’ day school if they have a question about their

progress or status.

The activities staff provide in the afterschool program are tied to specific learning goals

that are related to the school day curriculum.

Staff use student assessment data to provide different types of instruction to students

attending their afterschool activities based on their ability level.

Staff monitor students’ academic performance on district- or building-level assessments

across the school year and use this information to inform activities they provide.

Staff help manage a formal three-way communication system that links parents, program,

and day school information.


Staff participate in regular, joint staff meetings for afterschool and regular school day

staff where steps to further establish linkages between the school day and afterschool are

discussed.

Staff meet regularly with school day staff not working in the afterschool program to

review the academic progress of individual students.

Staff participate in parent-teacher conferences to provide information about how

individual students are faring in the afterschool program.

PROMPT: What strategies are used to link the program to the regular school day?

Align programming to school day curriculum and standards.

Help with homework.

Hire regular school day teachers.

Use student assessment and/or grades to inform programming.

Regular face-to-face meetings with school day staff.

Regular electronic communication with school day staff.

Regular electronic communications with principals and other school day administrative staff.

Regular monitoring of students’ academic performance on district- or building-level

assessments across the school year and use of this information to inform activity provision.

Ensure that activities are informed by and meant to support schoolwide improvement

targets related to student performance.

As shown in Table 21, the average site coordinator scale score fell within the minor strategy

response category, indicating that most sites employed only a portion of the listed strategies for

establishing linkages with the school day. As Figure 35a further shows, 53.22 percent of site

coordinator respondents fell within the minor strategy response category, and 31.58 percent fell

within the major strategy category. The least frequently adopted strategy was hiring regular

school day teachers to support links to the school day, and the most common strategy shifted

from align programming to school day curriculum and standards to help with homework. For

staff responses, the average scale score on linkages to the school day fell within the disagree

response category, suggesting that, on average, most staff have an incomplete sense of both

student academic needs and school day curriculum and/or instructions. As shown in Figure 36a,

30.94 percent of centers fell in the disagree response category, and 65.19 percent fell in the

agree category.

The average scale score on data use for both site coordinators and staff fell in the occasionally

use category, indicating the degree to which site coordinators and staff use data to inform

programming is still limited. As shown in Figure 35b and Figure 36b, 72.51 percent of site

coordinator respondents and 54.55 percent of staff responses fell in the occasionally use

category.

It is important to note that no effort was made to control for the staff person’s role in the

program, so staff responsible for the delivery of arts or recreation programming also were asked


questions related to links to the school day. Finally, responses to items related to the use of

student data to inform programming indicated that these practices were the least common

strategy used by staff to intentionally link programming to the school day. This finding is a

common finding among the evaluations conducted by the evaluation team. Here again, there may

be a role for local evaluators in both gathering and analyzing student data to better support

individual student needs.

Table 21. Statewide Performance on the Linkages to the School Day Scale

Statewide Mean

Range of Scale

Statewide Mean

Fell Within

Linkages to the School Day—Site Coordinator Survey The extent to which the site coordinator reports the

program taking steps to establish linkages to the school day

and using student data to inform programming.

60.55

(N = 186)

Minor Strategy

(N = 186)

Linkages to the School Day—Staff Survey The extent to which program staff report taking steps to

establish linkages to the school day and using student data

to inform programming.

58.66

(N = 1,038)

Disagree

(N = 1,038)

Data Use – Site Coordinator Survey The extent to which the site coordinator reports the

program using student data to inform programming.

60.47

(N = 186)

Occasionally Use

(N = 186)

Data Use – Staff Survey The extent to which program staff report taking steps to use

student data to inform programming.

57.94

(N = 915)

Occasionally Use

(N = 915)



Figure 35a. Summary of Responses Regarding Linkages With the School Day

From the Site Coordinator Survey


Figure 35b. Summary of Responses Regarding Data Use From the Site Coordinator

Survey


31.58%

53.22%

15.20%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Major Strategy Minor Strategy Not a Strategy

Sit

e C

oo

rdin

ato

rs

14.04%

72.51%

13.45%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

Do Not Receive Occasionally Use Often Use

Sit

e C

oo

rdin

ato

rs


Figure 36a. Center Classification Based on Staff Members’ Responses

Regarding Linkages to the School Day


Figure 36b. Center Classification Based on Staff Members’ Responses

Regarding Data Use


2.21%

30.94%

65.19%

1.66%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%


Cen

ters

22.16%

54.55%

23.30%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Do Not Receive Occasionally Use Often Use

Cen

ters


Indicator 3.3: Community Context

Encouraging partnerships between schools and community organizations is an important

component of the national 21st CCLC programs. Partners are defined as any organization other

than the grantee that actively contributes to a 21st CCLC-funded program to help programs meet

their goals and objectives. Partners may play a variety of roles in supporting a 21st CCLC-

funded program. For example, partners may provide programming and staff, provide physical

space and facilities, and facilitate fundraising efforts. In many instances, partners can play a

critical role in providing activities and services that the grantee lacks expertise or training in

to enhance the variety of learning opportunities available to youth.

From a quality perspective, mutually beneficial partnerships are most effective when staff from

the partner organization work directly with youth and are involved in regular program processes

related to staff orientation, training, evaluation, feedback, and professional development.

The leading indicator for community context is meant to capture the degree to which partners

associated with the center are actively involved in planning, decision making, evaluating, and

supporting program operations. Two separate metrics were calculated to describe aspects of this

indicator:

Partner Involvement: The extent to which partners associated with the center are

actively involved in planning, decision making, evaluating, and supporting the operations

of the afterschool program

Family and Community—YPQA Form B: The extent to which the program adopts

policies and practices supportive of family and community engagement

Partner Involvement

Scale scores for community context include site coordinator responses to the following survey

question:

PROMPT: Do you and representatives from partner agencies involved in afterschool

programming work together to do the following, and if you do, are these things done informally

or formally?

Establish goals and objectives for the program.

Orient new staff to the program.

Provide professional development opportunities to program staff.

Review evaluation results and target areas for improvement.

Develop and evaluate the effectiveness of operational procedures (e.g., recruitment,

scheduling, activity transitions, etc.).

Plan for program sustainability and/or expansion.

As shown in Table 22, the average statewide scale score on the partner involvement scale of the

site coordinator survey fell within the do informally response category (response options


included did not do, do informally, and do formally). Generally, while centers work with partners

in many of the ways described in the items appearing on this scale, they have a tendency to do so

on an informal basis as opposed to having formal policies and procedures in place to support

partnerships. Figure 37 shows that 18.81 percent of responding site coordinators fell within the

do formally response category, and 59.41 percent fell within the do informally response category.

It is also important to note that only 112 centers had partners that are actively involved in the

provision of programming directly to youth; this represents a little over half of the total centers

with site coordinator survey responses. This may indicate that more time and attention could be

dedicated to finding community partners that can enhance program activities through more direct

involvement in program provision.

Table 22. Statewide Performance on the Linkages to the Partner Involvement Scale

Statewide Mean

Range of Scale

Statewide Mean

Fell Within

Partner Involvement The extent to which partners associated with the center are

actively involved in planning, decision making, evaluating,

and supporting the operations of the afterschool program.

58.13

(N = 112)

Do informally

(N = 112)


Figure 37. Summary of Site Coordinator Responses Regarding Partner Engagement


18.81%

59.41%

21.78%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

Do Formally Do Informally Do Not Do

Sit

e C

oo

rdin

ato

rs


Family and Community

This indicator is meant to capture the barriers to participation and the linkages between program

and families and the community. As shown in Table 23, the statewide scale score on the family

and community fell within the three category. Figure 38 shows that 77.42 percent of staff

respondents fell in this category also. This indicates that a majority of staff reported the policies

and practices of the program promote family and community engagement.

Table 23. Statewide Performance on the Family and Community Scale

Statewide Mean

Range of Scale

Statewide Mean

Fell Within

Family and Community – YPQA Form B

The extent to which the program adopts policies and

practices supportive of family and community engagement.

52.42

(N = 62)

Three

(N = 62)


Figure 38. Summary of Site Coordinator Responses Regarding Family and Community


Summary of Findings and Recommendations in Relation to the Community

Context Domain

Of the domains represented in the leading indicator system, the indicators associated with the

Community Context domain are most likely to be influenced by local community contexts.

9.68%

77.42%

12.90%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

One Three Five

Sit

e C

oo

rdin

ato

rs


For example, Washington is characterized by a relatively large number of grantees that are not

school districts, and as a consequence, the mechanisms for developing effective relationships

with school day staff require both effort and a certain level of trust and rapport, which may have

a bearing on strategies for linking with the school day. In addition, Washington has a relatively

large number of grantees in rural settings (accounting for more than 40 percent of centers active

during the 2011–12 programming period), which can make the process of finding viable

partnerships more complicated, given the limited availability of community partners in rural

settings.

Of the indicators represented in the Community Context domain, it is the opinion of the

evaluation team that the School Context indicator is of greatest import for ensuring high-quality

21st CCLC programming aligned with goals of supporting student growth and development in

reading and mathematics. As with most indicators highlighted thus far in the report, there are

opportunities for growth in relation to establishing links to the school day, particularly in relation

to the use of student data to meet individual student needs. In order to better support grantees in

this regard, it is recommended that steps be taken to modify the leading indicator reports housed

in PPICS to make state assessment data available to grantees so that these data can be analyzed

to better identify student academic needs and inform the design and delivery of programming.

It also may make sense to make this part of the work done by the local evaluator as part of a

revised set of local evaluation guidelines, with guidance around approaches that should be

employed to examine the data and how results should be shared with program management. In

any event, one of the goals of the Year 3 evaluation is to modify the leading indicator reports

based on data associated with the 2012–13 programming period to include this functionality.

Youth Outcomes Leading Indicators

Indicator 4.1: Reading/ELA and Mathematics Performance

This leading indicator is meant to outline the extent to which students attending 21st CCLC

programming regularly (i.e., more than 30 days during the reporting period) moved from one

state proficiency category to another between the 2010–11 and 2011–12 school years in

reading/ELA and mathematics. Similar data are displayed for students attending programming

for fewer than 30 days and students who attended the same schools as 21st CCLC participants

but did not participate in programming. Each cell in Tables 24 and 25 represents the percentage

of students in a given group (i.e., regular attendee, non-regular attendee, or nonparticipant) who

scored in a particular proficiency category in 2010–11 that also scored in a particular proficiency

category in 2011–12. It is important that these data are descriptive in nature and cannot be used

to assess the program's causal impact on student outcomes in reading/ELA and mathematics.


Table 24. Mathematics Performance

2010–

11

2011–12

Level 1 –

Well Below

Standard

Level 2 –

Below

Standard

Level 3 –

Met

Standard

Level 4 –

Above

Standard

Center Data,

Regular

Attendees

Level 4 –

Above

Standard

1.52% 1.52%

Level 3 –

Met Standard 1.52% 3.03% 9.09%

Level 2 –

Below

Standard

10.61% 24.24% 7.58% 1.52%

Level 1 –

Well Below

Standard

25.76% 13.64%

Center Data,

Non-Regular

Attendees

Level 4 –

Above

Standard

Level 3 –

Met Standard 10%

Level 2 –

Below

Standard

20% 10% 10%

Level 1 –

Well Below

Standard

20% 20% 10%

Feeder School

Data

Level 4 –

Above

Standard

0.29% 2.92% 11.99%

Level 3 –

Met Standard 1.46% 6.14% 22.22% 9.06%

Level 2 –

Below

Standard

7.60% 6.73% 6.73% 0.29%

Level 1 –

Well Below

Standard

19.59% 3.51% 1.46%


Table 25. Reading/ELA Performance

2010–

11

2011–12

Level 1 –

Well Below

Standard

Level 2 –

Below

Standard

Level 3 –

Met

Standard

Level 4 –

Above

Standard

Center Data,

Regular

Attendees

Level 4 –

Above

Standard

1.61%

Level 3 –

Met Standard 9.68% 9.68% 3.23%

Level 2 –

Below

Standard

8.06% 32.26% 12.90% 4.84%

Level 1 –

Well Below

Standard

8.06% 8.06% 1.61%

Center Data,

Non-Regular

Attendees

Level 4 –

Above

Standard

Level 3 –

Met Standard 12.50%

Level 2 –

Below

Standard

25% 25% 25%

Level 1 –

Well Below

Standard

12.50%

Feeder School

Data

Level 4 –

Above

Standard

1.20% 8.71% 16.22%

Level 3 –

Met Standard 0.60% 4.80% 16.52% 15.92%

Level 2 –

Below

Standard

2.70% 10.51% 6.01% 2.40%

Level 1 –

Well Below

Standard

12.01% 2.10%

0.30%


Determining Program Improvement Priorities From the

Leading Indicator System

One of the goals of the leading indicator system is to help OSPI make a determination regarding

where efforts should be invested to support programs in the adoption of quality afterschool

practices. For each scale represented in the leading indicator system, a portion of that scale

indicates that a quality approach or practice is largely not being adopted by the center in

question. In Table 26, each of the indicators and related scales are listed along with the portion of

the scale that indicates that a given practice is largely absent from the center in question and the

number and percentage of centers that fall within these ranges. As shown in Table 26, there is no

scale in which more than 50 percent of centers fell within a range indicating that the quality

practice was largely absent, in comparison with the result in the 2010–11 evaluation where there

were two such scales: the youth ownership scale from the site coordinator survey and the

linkages to the school day scale from the staff survey. This change may indicate improvement in

the quality practice of the two indicators.

In addition, it should be noted that there were differences in terms of scales where over 20 percent

of centers reported significant lack of quality practice between 2010–11 and 2011–12

programming periods. As indicated in Table 27, more efforts are still needed for the development

of program climate, youth ownership, and linkages to the school day. In addition, it may be

worthwhile to consider how to promote staff’s data use and partner involvement in program

support.

In terms of climate, our sense is that a lack of time for session planning and preparation is a

substantive issue in these centers, and that it may be worthwhile for OSPI to consider what

message it wants to send to grantees regarding the importance of creating space for lesson planning

and session preparation and how this message could be sent, both in terms of how RFPs are crafted

and the types of professional development and tools provided to project directors and center

coordinators.

As already noted, the linkages to the school day scale was calculated based on all staff members

responding to the survey regardless of staff roles. Staff role relative to providing activities with

an explicit academic focus is being tracked in a more intentional way as part of the survey data

collection process for 2012–13, giving the evaluation team a way to examine adoption of these

practices more thoroughly, based on the staff member’s role in the program. As a result, it is our

sense that it would be premature to draw overly definitive conclusions about staff practice in this

area.

In terms of the youth ownership scale, more definitive conclusions can be reached about the

percentage of centers where these practices are largely absent, and findings from the staff survey

reinforce these results as well as those shown in Table 26. The bigger question here is how to

cultivate center adoption of these practices in a developmentally appropriate fashion that takes

into consideration the diversity of grade levels served by 21st CCLC programs in Washington.

This is a topic that warrants additional consideration during future years of the evaluation. In

terms of data use, our sense is that it may be worthwhile to consider promoting the program’s

access to individual education plans and students’ scores on district and state level assessments,

and engaging staff in using this information to plan activities.


Table 26. Leading Indicator Scales by Number and Percentage of Centers Where Quality

Practices Were Largely Absent

Domain/Scale

Rating Options

Indicating Practice

Not Present

N

Centers

%

Centers

Organization Context

Program Climate The extent to which program staff report that a

supportive and collaborative climate exists within the

program

Disagree, Strongly

Disagree 39 21.67%

Internal Communication—Site Coordinator

Survey The frequency with which the site coordinator engages in

practices with program staff that support internal

communication and collaboration

Never 4 2.35%

Internal Communication—Staff Survey The frequency with which the staff engages in

practices with other program staff that support internal

communication and collaboration

Never 6 3.31%

Leadership and Management—YPQA Form B

The extent to which the program is engaging in

practices that ensure staff are well positioned to

create developmentally appropriate settings for youth

and that processes are in place to support program

improvement efforts

Five 6 9.68%


Program Objectives—The degree to which site

coordinators’ top three objectives align with proportion

of time spent on corresponding activities

Alignment not present 19 10%

Intentionality in Program Design—Site

Coordinator Survey The frequency with which staff engage in practices

that indicate intentionality in activity and session

design among staff responsible for the delivery of

activities meant to support student growth and

development.

Rarely 14 8.24%

Intentionality in Program Design—Staff Survey The frequency with which staff engage in practices

that indicate intentionality in activity and session

design among staff responsible for the delivery of

activities meant to support student growth and

development.

Rarely 4 2.35%


Domain/Scale

Rating Options

Indicating Practice

Not Present

N

Centers

%

Centers

Point of Service Quality—YPQA Form A

The extent to which program staff provide supports

and opportunities to create safe, supportive,

interactive, and engaging settings for participating

youth (total YPQA score)

Still Room for

Improvement 14 19.44%

Safe Environment 0 0

Supportive Environment Still Room for

Improvement 8 11.11%

Interaction Functioning Near

Optimal 12 16.67%

Engagement Functioning Near

Optimal 7 9.72%

Youth-Centered Policies & Practices—YPQA

Form B

The extent to which the program adopts youth-

centered policies and practices conducive to a

supportive learning environment

One 6 9.68%

Youth Ownership—Site Coordinator Survey The extent to which the site coordinator perceives

program staff extending opportunities to youth to

develop ownership in the program

Disagree, Strongly

Disagree 81 47.65%

Youth Ownership—Staff Survey The extent to which program staff perceive staff

members extending opportunities to youth to develop

ownership in the program

Disagree, Strongly

Disagree 69 38.13%

Capacity to Create Interactive and Engaging

Settings—The extent to which staff members perceive

program staff extending opportunities and providing

supports to youth that result in the creation of an

interactive and engaging setting for participating youth

Disagree 21 12.35%

Service Delivery Practices—The frequency with

which staff adopt practices that support youth

development

Never 1 0.55%


Family Communication—The frequency with which

staff adopt practices that support communication with

parents and adult family members

Never 20 11.76%


Domain/Scale

Rating Options

Indicating Practice

Not Present

N

Centers

%

Centers

Linkages to the School Day—Site Coordinator

Survey The extent to which the site coordinator reports the

program taking steps to establish linkages to the

school day and using student data to inform

programming

Not a strategy 26 15.2%

Linkages to the School Day—Staff Survey The extent to which program staff report taking steps

to establish linkages to the school day and using

student data to inform programming

Disagree, Strongly

Disagree 60 33.15%

Data Use—Site Coordinator Survey

The extent to which the site coordinator reports the

program using student data to inform programming

Do Not Receive 24 14.04%

Data Use—Staff Survey

The extent to which program staff report taking steps

to use student data to inform programming

Do Not Receive 39 22.16%

Partner Involvement The extent to which partners associated with the center

are actively involved in planning, decision making,

evaluating, and supporting the operations of the

afterschool program

Do not do 22 21.78%

Family and Community – Form B PQA

The extent to which the program adopts policies and

practices supportive of family and community

engagement

One 6 9.68%


Table 27. Scales Where More Than 20 Percent (Including More Than 50 Percent) of

Centers Report Lack of Quality Practice

Scale 2010–11 School Year

(% Centers)

2011–12 School Year

(% Centers)

Program climate √

(27%)

√

(22%)

Capacity to create interactive and

engaging settings

√

(26%)

Program objectives √

(22%)

Youth ownership—site coordinator √

(56%)

√

(48%)

Youth ownership—staff survey √

(48%)

√

(38%)

Linkages to the school day—staff survey √

(52%)

√

(33%)

Data use—staff survey √

(22%)

Partner involvement √

(22%)



Chapter 5: Assessing 21st CCLC Program Outcomes

Within-Program and Impact Analyses

Another primary objective of the statewide evaluation was to understand the relationship

between participation in 21st CCLC-funded programs and student academic behaviors and

outcomes. Employing program participation and outcome data associated with the 2011–12

programming period, two analytic approaches were used:

Within-program analyses. The within-program analyses examined the relationship

between student outcomes and several student and program characteristics, with a

particular emphasis on exploring the relationship between leading indicator status and

youth outcomes. The analyses are correlational in nature, meaning that inferences about

causation or directionality cannot be made. Other factors that were not included in the

analyses may play a role in the reported findings.

Impact analyses. The impact analyses were based on a rigorous quasi-experimental

design that compared academic outcomes of 21st CCLC program participants with

matched nonparticipating students using a propensity score matching approach.

Meaningful conclusions may be drawn from the impact analysis regarding the impact of

Washington 21st CCLC program participation on student outcomes.

To determine student- and center-level characteristics related to the student outcomes under

consideration, the evaluation team employed a series of hierarchical linear models (HLMs) to

test for statistically significant relationships between student and center characteristics and a

variety of youth outcomes. Findings from these analyses are described in the following section.

Within-Program Analyses

Given the time and effort involved in both collecting and analyzing the data needed to support

population of the leading indicators, it seemed appropriate to explore if a center’s status on a

given set of leading indicators was related to (a) the degree to which youth participated in the

21st CCLC program during the 2011–12 project period and (b) performance on school-related

outcomes. In order to both simplify these analyses and triangulate data from multiple leading

indicators related to a given domain of center performance, hierarchical cluster analysis was used

to classify centers into one of three primary subgroups based on the center’s performance across

indicators in that domain:

When all indicators suggest above average performance in relation to a given

quality domain. These are centers that may warrant further examination to learn more

about the strategies that support effective implementation of quality practices.

It would also be expected that the likelihood that such centers would have a positive

impact on student outcomes would be greater.

When all indicators suggest below average performance in relation to a given

quality domain. These are centers that could especially benefit from additional services

and supports to enhance the quality of program implementation. Similar to the

information presented in Table 26, knowing how many centers fall within this category


across the various quality domains could prove useful to OSPI as it structures and

prioritizes its technical assistance and training agenda.

Mismatches in indicators in relation to a given quality element. These are centers in

which there is divergence in the indicators of implementation within a given domain.

These mismatches may suggest a lack of communication and shared vision and

understanding among key actors within the program. In these centers, consideration could

be given to achieving a shared vision and understanding of the goals, planning

requirements, implementation characteristics (e.g., high-level planning and management

and day-to-day tasks), program improvement strategies, challenges, and data/outcomes

associated with effective implementation of 21st CCLC programming.

Hierarchical cluster analyses were performed in relation to three leading indicator domains:

(1) Organizational Context, (2) Instructional Context-Content, and (3) Instructional Context-

Process. The next section summarizes results from each cluster analysis, and then steps are taken

to highlight findings that summarize the relationship between cluster membership status and

program attendance and school-related outcomes.

Organizational Context Clusters

Center performance on four indicators associated with the Organizational Context domain was

used to form clusters in this domain.2

Training Participation

Program Climate

Site Coordinator-Reported Internal Communication

Staff-Reported Internal Communication

As shown in in Figure 39, application of hierarchical cluster analysis techniques yielded the

following three quality profiles:

1. Site Coordinator Internal Communication Score Especially Above Average. A total

of 49 centers (29 percent of the total with complete indicator data) were assigned to this

cluster where scores on three of the four leading indicators under consideration were

above average, particularly the site coordinator’s score on the internal communication

scale. Centers in this cluster would be considered to have a higher degree of

implementation on strategies and practices that support staff development and internal

communication and collaboration among program staff.

2. All Indicators Below Average. A total of 92 centers (54 percent of the total with

complete indicator data) were assigned to this cluster where scores on all four leading

indicators under consideration were below average. Centers in this cluster would be

considered to have a lower degree of implementation on strategies and practices that

2 It is important to note that indicators predicated on YPQA data were not included in any of the hierarchical cluster

analyses detailed in this section of the report. Hierarchical cluster analysis requires complete data across all fields

included in the analysis. Including YPQA data would have greatly reduced the number of centers that could be

included in these analyses because only a portion of the 21st CCLC population of centers participated in the YPQI

process during 2011–12.


support staff development and internal communication and collaboration among program

staff.

3. Staff Climate and Internal Communication Scores Above Average. A total of 28

centers (17 percent of the total with complete indicator data) were assigned to this cluster

where scores on indicators based on data collected from activity leaders working in 21st

CCLC programs were especially above average, while the average site coordinator score

on the internal communication scale was below average. This disconnect between staff

and site coordinator scores among centers in this cluster is of particular interest, possibly

demonstrating more staff-to-staff interaction and communication than similar processes

occurring between the site coordinator and staff.

Figure 39. Clusters Related to Strategies and Practices That Support a High-Quality


Source: 169 centers with data on leading indicators related to Organizational Context

Instructional Context-Content Clusters

Center performance on six indicators associated with the Instructional Context-Content domain

was used to form clusters in this domain:

Site Coordinator-Reported Intentionality in Program Design

Staff-Reported Intentionality in Program Design

Site Coordinator-Reported Linkages to the School Day

Staff-Reported Linkages to the School Day

Site Coordinator-Reported Use of Student Data to Inform Programming

-1.5

-1

-0.5

0

0.5

1

1.5

SC comm above

average (29%)

All below average

(54%)

Staff climate and

comm above

average (17%)

Mea

n S

tan

da

rdiz

ed S

core

Mean Training Attendance

Mean Climate Score

Mean SC Internal Comm Score

Mean Staff Internal Comm Score


Staff-Reported Use of Student Data to Inform Programming

As shown in in Figure 40, application of hierarchical cluster analysis techniques yielded three

quality profiles:

1. Site Coordinator Scores Above Average. A total of 75 centers (46 percent of the total

with complete indicator data) were assigned to this cluster where scores on three of the

six leading indicators under consideration were noticeably above average, all of which

pertained to the site coordinator’s perception on indicators related to the Instructional

Context-Content domain. This may be an indication that ownership of practices

associated with the alignment of programming to school-related considerations may

reside primarily with site coordinators in this cluster, as compared with a more shared

approach where frontline delivery staff share significant responsibility in engaging in

such practices and approaches to support the content delivery objectives associated with

the center’s program.

2. All Indicators Above Average. A total of 21 centers (13 percent of the total with

complete indicator data) were assigned to this cluster where scores on all six leading

indicators under consideration were above average. Generally, centers in this cluster

would be considered to have a higher degree of implementation on strategies and

practices that support the embedding of academic content into program activities aligned

with the school day.


complete indicator data) were assigned to this cluster where scores on all six leading

indicators under consideration were below average. This was particularly the case in

relation to indicators based on site coordinators’ perceptions of linkages with the school

day and use of school-related data to inform the design and delivery of programs.



Instructional Context-Content

Source: 163 centers with data on leading indicators related to Instructional Context-Content

Instructional Context-Process Clusters

Center performance on four indicators associated with the Instructional Context-Process domain

was used to form four clusters in this domain:

Site Coordinator-Reported Opportunities for Youth Ownership

Staff-Reported Opportunities for Youth Ownership

Site Coordinator-Reported Efficacy of Staff in Creating Interactive and Engaging

Settings

Staff-Reported Use of Youth Development Practices

As shown in in Figure 41, application of hierarchical cluster analysis techniques yielded three

quality profiles:

1. Site Coordinator Scores Above Average. A total of 100 centers (59 percent of the total

with complete indicator data) were assigned to this cluster where scores on two of the

four leading indicators under consideration were noticeably above average, all of which

pertained to the site coordinator’s perception on indictors related to the Instructional

Context-Process domain. This may be an indication that ownership of practices

associated with the provision of youth ownership opportunities may reside at the program

level through opportunities such as youth program governance as opposed to being a core

aspect of activity delivery at the point of service. Site coordinators in this cluster also

believed their staff members were engaging in practices that would create an interactive

and engaging environment for participating youth.

-1.5

-1

-0.5

0

0.5

1

1.5

SC above average

(46%)

All above average

(13%)

All below average

(41%)

Mea

n S

tam

da

rdiz

ed S

core

Mean SC design score

Mean staff design score

Mean SC linkages score

Mean staff linkages score

Mean SC data use score

Mean staff data use score


2. Staff Scores Above Average. A total of 43 centers (25 percent of the total with complete

indicator data) were assigned to this cluster where scores on two of the four leading

indicators under consideration were above average, although in this case the indicators

well above average were associated with the staff survey. This result is somewhat of a

curious one, suggesting that staff members are either overestimating the provision of such

opportunities to youth in their own practice or that site coordinators in such centers are

largely unaware of what their staff are doing in regards to engaging in practices and

approaches designed to support youth development and ownership in the program.


complete indicator data) were assigned to this cluster where scores on all four leading

indicators under consideration were below average. This was particularly the case in

relation to indicators based on staff perceptions of the provision of opportunities for

youth development and ownership.


Instructional Context-Process

Source: 169 centers with data on leading indicators related to Instructional Context-Process

Primary Hypothesis Tested Through Within-Program Analyses

The within-program analyses highlighted in the sections that follow were primarily oriented at

exploring the veracity of one primary hypothesis related to the assignment of centers to different

clusters constructed from the leading indicators by domain:

-1.5

-1

-0.5

0

0.5

1

1.5

SC above average

(59%)

Staff above average

(25%)

All below average

(15%)

Mea

n S

tan

da

rdiz

ed S

core

Mean SC onwnership scale

Mean staff ownership scale

Mean SC efficacy scale

Mean staff YD practices scale


It was hypothesized that there would be a negative correlation between center

membership in the All Indicators Below Average cluster for each domain and youth

program attendance and outcomes.

As noted in the prior sections, each domain was found to have one cluster where centers were

below average on each of the indicators represented in that domain, although as shown in Figures

39–40, the percentage of centers falling in the All Indicators Below Average cluster varied

significantly by domain:

Organizational Context: 54 percent of centers fell in the All Indicators Below Average

cluster.

Instructional Context-Content: 41 percent of center fell in the All Indicators Below

Average cluster.

Instructional Context-Process: 15 percent of center fell in the All Indicators Below

Average cluster.

A decision was made to focus on centers classified in the All Indicators Below Average cluster

based on an assumption that issues of social desirability in response patterns would be less of an

issue in relation to scores associated with centers in these clusters, and consequentially, estimates

derived from site coordinator and staff surveys may be more indicative of the actual level of

functioning at these centers. In this sense, it was expected that there would be a negative

relationship between membership in an All Indicators Below Average cluster and the attendance

school-related outcomes examined.

Summary of Within-Program Analysis Variables

In addition to whether or not a given center fell in an All Indicators Below Average cluster, the

multilevel models underpinning the within-program analyses explored associations between a

variety of student- and center-level characteristics and student outcomes for 21st CCLC

participants. Student outcomes included the following:

Youth participation in the 21st CCLC program during the 2011–12 project period (all

grades)

2012 Measurements of Student Progress (MSP) reading assessment performance

(Grades 4–8)

2012 MSP mathematics assessment performance (Grades 4–8)

2012 High School Proficiency Exam (HSPE) reading assessment performance (Grade 10)

Cumulative grade point average (GPA) (Grades 9–12)

Percentage of credits earned relative to credits attempted (Grades 9–12)

Number of unexcused absences for the 2011–12 school year (all grades)

In addition to the aforementioned youth outcomes, a variety of other student-level characteristics

were included in the within-program models:

Youth grade level


Status as a racial minority

Hispanic ethnicity

Gender

Eligibility for free or reduced-price lunch

Special education status

Limited English proficiency status

A summary of youth characteristics across each of these areas is displayed in Table 28. As

shown, most youth included in the within-program analyses were in Grades 4–8 (75 percent), the

majority were identified as racial minorities (71 percent), there were slightly more males than

females (51versus 49 percent, respectively), about four fifths (83 percent) qualified for free or

reduced-price lunch, 20 percent were designated as having limited proficiency in English, and 14

percent were receiving special education services.

Table 28. Summary Statistics: Student Characteristics

Proportion of 2011–12

21st CCLC Participants

Grade Level (n = 14,925)

3 0.099

4 0.105

5 0.104

6 0.196

7 0.192

8 0.152

9 0.048

10 0.059

11 0.030

12 0.016

Minority Status (n = 14,900)

Minority 0.707

Nonminority 0.293

Hispanic Status (n = 14,900)

Yes 0.398

No 0.602



21st CCLC Participants

Gender (n =14,925)

Male 0.514

Female 0.486

Free or Reduced-Price Lunch Status (n = 14,871)

Eligible 0.826

Not eligible 0.174

Special Education Status (n = 14,871)

Yes 0.140

No 0.860

Limited English Proficiency (n = 14,871)

Yes 0.200

No 0.800

As shown in Table 29, in addition to whether or not a given center fell in an All Indicators Below

Average cluster, other center-level characteristics included in the within-program models

included the following:

Whether or not the center was associated with a school-based grantee (45 percent)

Whether or not the center was staffed mostly by school day teachers (45 percent)

Whether or not the site coordinator had at least a bachelor’s degree (77 percent)

These center-level characteristics were found to be significant in within-program models run in

relation to data associated with the 2010–11 program period and, therefore, were considered

viable candidates for inclusion in the analyses undertaken in relation to 2011–12.

It is also important to note that HLM requires complete data at the center level for all variables

included in the model. In this regard, although cluster assignment information was available for

169 centers for the Organizational Context and the Instructional Context-Process domain, only

163 centers had data in relation to the Instructional Context-Content domain. As a result, 163

centers were included in the within-program analyses for 2011–12.


Table 29. Summary Statistics: Center Characteristics


21st CCLC Centers

Grantee Type (n = 163)

School-based 0.454

Non-school-based 0.546

Staffing Cluster (n = 163)

Mostly teachers 0.454

All other staffing clusters 0.546

Site Coordinator has bachelor degree or above (163)

Yes 0.773

No 0.227

Leading Indicator Cluster: Organizational Context Below Average (n = 163)

Yes 0.467

No 0.534

Leading Indicator Cluster: Instructional Context-Content Below Average (n = 163)

Yes 0.411

No 0.589

Leading Indicator Cluster: Instructional Context-Process Below Average (n = 163)

Yes 0.147

No 0.853

Within-Program Analysis Results and 21st CCLC Program Attendance

Results outlining the relationship between center and youth characteristics and the number of

days attending 21st CCLC programming during the 2011–12 project period are outlined in Table

30.

Again, the primary goal of the within-program analysis was to explore if a relationship could be

found between membership in a cluster indicating below average levels of implementation on the

practices associated with a particular leading indicator domain and youth outcomes. As shown in

Table 30, a significant, negative relationship was found between center membership in the

Organizational Context Below Average cluster and youth attendance in 21st CCLC. This

relationship is consistent with our hypothesis, indicating that 21st CCLC attendance was lower in

centers where adoption of practices associated with the Organizational Context domain was

below average.

Contrary to our hypothesis, however, center membership in the Instructional Context Process

Below Average cluster was found to be significantly and positively related to 21st CCLC

attendance, indicating a higher level of attendance in centers falling into this cluster. Although

initially puzzling, a further examination of the within-program analysis data set demonstrated

that centers falling in the Instructional Context Process Below Average cluster were much more

likely to be centers that served elementary students only (p < .01). It is a well-established finding


that the total number of days of attendance in 21st CCLC programming is substantially higher in

elementary programs compared with programs serving middle or high school students. In fact,

this finding is also present in Table 30: Middle and high school youth were found to have

significantly lower levels of attendance in 21st CCLC compared with elementary students. It is

also known that elementary programs are less apt to score as well on the YPQA, which contains

many of the youth-development-related practices represented in the Instructional Context

Process Below Average cluster. It is possible that these core grade-level differences associated

with membership in the Instructional Context Process Below Average cluster are associated with

the positive relationship observed in Table 30.

Other significant findings outlined in Table 30 include the following:

Grant School-Based (Negative Relationship). A negative relationship was found to

exist between a center’s status as being associated with a school-based grantee and youth

attendance in the 21st CCLC program. In this sense, 21st CCLC attendance was found to

be lower in centers run by school-based grantees.

Continuous Years of Enrollment (Positive Relationship). Youth enrolled in 21st

CCLC for two or more years were more likely to have higher program attendance in

2011–12.

Table 30. Model Results: Program Attendance Outcome With

Leading Indicator Predictors

Predictors 21st CCLC Program Attendance

Grant school-based -0.128**

(0.054)

Mostly teachers 0.012

(0.054)

Academic site coordinator has B.A.+ -0.006

(0.062)

Organizational Context below average -0.181***

(0.054)

Instructional Context-Content below average -0.036

(0.053)

Instructional Context-Process below average 0.127*

(0.077)

Slopes

Continuous years 0.192***

(0.010)

Middle school student -0.309***

(0.027)

High school student -0.312***

(0.055)

Free or reduced-price lunch eligible -0.005

(0.013)

Special Education 0.025*

(0.013)

Note. Standard errors are reported in parentheses; *** sig. at 0.01; ** sig. at 0.05;

* sig. at 0.10.


Within-Program Analysis Results and Unexcused Absences

Results outlining the relationship between center and youth characteristics and the number of

unexcused school-day absences during the 2011–12 school year are outlined in Table 31. Given

that unexcused school-day absences are a negative outcome, it was expected that we would see a

positive relationship between membership in a cluster defined by below-average performance on

the leading indicators and the number of unexcused absences accumulated by program youth.

As shown in Table 31, this was found to the case in relation to membership in the Instructional

Context-Content Below Average cluster. In this case, a significant positive relationship was

found between membership in the Instructional Context-Content Below Average cluster and the

number of unexcused school-day absences accumulated by participating youth. Although

consistent with our overall expectation that membership in a below-average cluster would be

associated with less desirable youth outcomes, we considered it more likely that membership in

the Instructional Context Process Below Average cluster would be more apt to be associated with

higher school-day unexcused absences because the youth development aspects of the

programming would help draw and retain students in the 21st CCLC program and by proxy

support school-day attendance. It may be the case that a focus on content also may be associated

with more explicit targeting of youth for participation in programming, which may mean more

communication with parents and guardians, which in turn may facilitate both program and

school-day attendance. At this point, such possible explanations are mere speculation.

In terms of other findings outlined in Table 31, a number of youth characteristics were associated

with higher rates of unexcused absences, including (a) high school students, (b) youth eligible for

free or reduced-price lunch, (c) special education students, and (d) minority youth. Youth

characteristics negatively associated with unexcused absences included (a) Hispanic youth and

(b) students with higher levels of attendance in 21st CCLC. The latter result will be examined

more closely in the impact analysis section of this report where results for 21st CCLC

participants versus non-participating youth attending the same schools are examined.

Finally, youth associated with centers associated with school-based grantees also had higher

levels of unexcused absences than youth attending centers run by non-school-based grantees.

Table 31. Model Results: Unexcused Absences Outcome With

Leading Indicator Predictors

Predictors Number of Unexcused Absences

Grant school-based 0.689**

(0.280)

Mostly teachers -0.065

(0.284)

Academic site coordinator has B.A.+ -0.223

(0.323)

Organizational Context below average 0.209

(0.283)

Instructional Context-Content below average 0.488*

(0.279)

Instructional Context-Process below average -0.258

(0.409)


Slopes

SY days -0.0060***

(0.001)

Continuous years 0.004

(0.032)

Middle school student -0.113

(0.097)

High school student 0.667***

(0.145)

Free or reduced-price lunch eligible 0.540***

(0.047)

Special education 0.220***

(0.040)

Limited-English-proficient status -0.019

(0.039)

Gender (1 = male) 0.037

(0.029)

Hispanic -0.095**

(0.043)

Minority 0.187***

(0.049)

Notes. Standard errors are reported in parentheses; *** sig. at 0.01; ** sig. at 0.05; * sig. at 0.10.

Within-Program Analysis Results and Academic-Performance-Related Outcomes

Results outlining the relationship between center and youth characteristics and academic

performance-related outcomes associated with the 2011–12 school year are outlined in Table 32.

Outcomes related to academic performance include performance on state assessments in reading

and mathematics (primarily MSP, but also HSPE for Grade 10 reading), cumulative GPA, and

percentage of credits earned.

As shown in Table 32, the only below-average cluster predicated on leading indicator data

negatively associated with the academic-performance-related outcomes examined as part of the

within-program analyses was membership in the Instructional Context Content Below Average

cluster and reading scores. In this sense, students attending centers characterized by below-

average performance on the leading indicators related to the Instructional Context-Content

domain scored significantly lower on reading state assessments taken during the 2011–12 school

year.

The only other result related to cluster membership and academic-performance-related outcomes

pertains to a positive relationship between membership in the Organizational Context Below

Average cluster and both cumulative GPA and credits earned. Because both of these outcomes

pertain to high school students specifically, our sense is that the nature of this unexpected

relationship somehow relates to some element of the operation of high school programs.

In terms of other outcomes, across all academic performance-related outcomes, a significant

positive relationship was found to exist between days of participation in 21st CCLC

programming and academic-performance-related outcomes. The more youth attended

programming, the better they were found to perform academically. In addition, youth attending


the program for two years or more also were shown to score better on the mathematics portion of

the state assessments.

Yet again, youth associated with centers associated with school-based grantees were found to

demonstrate lower levels of performance on both mathematics and reading assessments, and

youth attending centers staffed mostly by school-day teachers were found to perform better on

mathematics state assessments.

Finally, a host of different student characteristics were found to be related to academic-

performance-related outcomes. Youth who were eligible for free or reduced-price lunch, were

receiving special education services, were limited English proficient, were male, and were

minorities were found to perform lower on one or more academic-performance-related outcomes.

Table 32. Model Results: Achievement Outcomes With Leading Indicator Predictors

Predictors Mathematics Reading/

Language Arts

Cumulative

GPA

Percent of Credits

Earned

Grant school-based -0.074**

(0.033)

-0.046*

(0.026)

-0.181

(0.167)

-0.055

(0.034)

Mostly teachers 0.062*

(0.033)

-0.013

(0.025)

-0.034

(0.192)

-0.025

(0.038)

Academic Site Coordinator has

B.A.+

0.009

(0.038)

0.005

(0.029)

0.017

(0.196)

0.025

(0.039)

Organizational Context below

average

0.025

(0.033)

0.005

(0.025)

0.295*

(0.177)

0.073**

(0.034)

Instructional Context-Content

below average

-0.029

(0.033)

-0.043*

(0.025)

0.022

(0.192)

0.002

(0.038)

Instructional Context-Process

below average

0.072

(0.047)

0.047

(0.036)

-0.572

(0.357)

-0.052

(0.069)

Slopes

2011 standardized score 0.728***

(0.007)

0.722***

(0.007) - -

SY Days 0.001***

(0.0002)

0.001***

(0.0002)

0.004***

(0.001)

0.001***

(0.0002)

Continuous years 0.034**

(0.013)

0.017

(0.014)

-0.041

(0.041)

0.001

(0.010)

Middle school student 0.052*

(0.028)

0.056**

(0.027) - -

High school student - -0.147

(0.269) - -

Free or reduced-price lunch

eligible

-0.081***

(0.017)

-0.082***

(0.017)

-0.257***

(0.054)

-0.056***

(0.013)

Special education -0.252***

(0.018)

-0.287***

(0.019)

-0.151**

(0.063)

0.013

(0.015)

Limited-English-proficient status -0.128***

(0.017)

-0.157***

(0.018)

0.038

(0.053)

0.015

(0.013)

Gender (1 = male) 0.008

(0.012)

-0.035***

(0.012)

-0.279***

(0.038)

-0.051***

(0.009)

Hispanic -0.031

(0.020)

0.004

(0.020)

0.073

(0.054)

0.020

(0.013)

Minority -0.010

(0.020)

-0.034*

(0.020)

-0.146***

(0.056)

-0.025*

(0.013)

Notes. Standard errors are reported in parentheses; *** sig. at 0.01; ** sig. at 0.05; * sig. at 0.10.


Summary of Within-Program Analyses Findings

The primary goal of the within-program analyses was to explore if there was evidence to support

the hypothesis that there would be a negative correlation between center membership in the all

indicators below average cluster for each domain and youth program attendance and outcomes.

The most consistent evidence to support this hypothesis was found in relation to center

membership in the Instructional Context-Content Below Average cluster where the hypothesized

relationship was found to exist in relation to the number of unexcused absences and performance

in reading state assessments.

In addition, membership in the Organizational Context Below Average cluster was also found to

be negatively associated with youth attendance in the program as anticipated, but positively

associated with cumulative GPA and credits earned.

Similar unexpected findings were found in relation to membership in the Instructional Context

Process Below Average and 21st CCLC program attendance where membership in this cluster

was associated with higher levels of program attendance, although it hypothesized that this result

may be driven by the high number of centers serving elementary only represented in this cluster.

Generally, the evidence supporting our hypothesis on the relationship between cluster

membership based on leading indicator performance and youth outcomes was not found to be

consistent or pervasive across all outcomes examined. The most consistent findings were in

relation to the Instructional Context Content Below Average cluster, with similar analyses

conducted as part of the statewide 21st CCLC evaluation in New Jersey demonstrating a similar

connection between content and youth outcomes. Given the percentage of centers falling in the

below-average cluster (41 percent), there are also opportunities to support further growth in

performance on these indicators through the expansion of training and technical assistance

efforts.

Also worthy of note is the finding that a significant, positive relationship was found between the

number of days of 21st CCLC participation and each of the academic-performance-related

outcomes examined. This is the type of relationship one would like to see if one were looking for

evidence that participation in 21st CCLC may be having a positive impact on such outcomes.

Finally, it is important to keep in mind the domain of within-program analyses conducted here

are correlational and descriptive in nature and do not permit causal inferences. For example, the

within-program findings cannot answer the question on whether more days of program

participation caused students to score higher on achievement tests. A correlational finding

between more days of program attendance and higher student achievement may instead explain

the characteristics of participating students. A correlation may exist because students who enjoy

school may be more likely to achieve higher assessment scores, and students who enjoy school

may be more likely to participate in programming that is similar to their school day activities—

that is, they may have higher levels of attendance in the 21st CCLC programs.

Taken together, the findings for within-program analyses are useful in exploring particular

student or center characteristics associated with lower (or higher) levels of student academic and


behavioral outcomes. The reader should keep in mind that these findings are purely descriptive

in nature and do not in any way imply that a given center or student characteristic is causally

related to a given outcome.

Impact of 21st CCLC Participation on Student Achievement

Although the within-program analyses were correlational in nature, steps were taken to also run a

series of causal models to assess the impact of the 21st CCLC on a variety of youth outcomes in

2011–12. In order to construct such causal estimates, the evaluation team employed a quasi-

experimental research design to examine the effect of participating in 21st CCLC programming

on student reading and mathematics achievement; cumulative GPA; percent of credits earned;

and one nonacademic measure: number of unexcused absences. Students’ reading and math

achievement were measured by the Washington state exam for Grades 3–8, the Measurements of

Student Progress (MSP) and the state exam for high school students, High School Proficiency

Exam (HSPE). The goal of this analysis was to answer the following evaluation questions:

To what extent is there evidence that students participating in services and activities

funded by 21st CCLC demonstrated better performance on reading and mathematics

assessments, cumulative GPA, and percentage of credits earned as compared with similar

students not participating in the program?

To what extent is there evidence that there are differences between students participating

in services and activities funded by 21st CCLC and similar students not participating in

the program in terms of the number of unexcused absences?

Specifically, using a propensity score stratification approach, the study compared the

performance of students who participated in 21st CCLC with similar students who did not

participate. Participation was defined two ways for the purpose of the analysis. First, students

who attended at least 30 days were compared with students who attended 0 days. Second,

students who attended at least 60 days were compared with students who attended 0 days. These

definitions of treatment were determined to ensure that the comparison of program effect was

based on students who received a significant dose of 21st CCLC programming.

In any evaluation of a program where participants are not randomly assigned to participate in the

program, the problem of selection is paramount. We know that it is likely that students who

participate in 21st CCLC programming are different from those who do not attend. These

differences can bias estimates of program effectiveness because they make it difficult to

disentangle preexisting differences between students who attended the program and those who

did not, from the effect of attending the program. In general, we found that students who

attended the program tended to be lower achieving students than those who did not, prior to the

start of the current academic year. The quasi-experimental approach outlined here, propensity

score matching (PSM), is a method for mitigating the existing bias in program effect (i.e., if one

were to simply compare the students who attended and those who did not).

PSM is a two-stage process designed to address this problem. In the first stage, the probability

that each student participates in the 21st CCLC program was modeled on available observable

characteristics. By modeling selection into the program, this approach allowed us to compare

participating and nonparticipating students who would have had a similar propensity to select


into the program based on observable characteristics that were available in the data received

from the state of Washington. In the second stage, the predicted probability of participation was

used to model student outcomes while accounting for selection bias. We balanced pretreatment

group differences in observed covariates using a propensity score stratification and marginal

mean weighting approach (Hong & Hong, 2009).

Stage 1: Creation of the Comparison Group

The outcome of interest in modeling propensity scores is treatment status (1 for students

participating in the program, 0 for the comparison group). To account for this binary outcome,

logistic regression was used to model the logit (or log-odds) of student group assignment status.

Examples of student-level variables used to fit the propensity score models included:

Prior achievement in reading and math

Student demographic information including

Gender

Racial status

Language of origin

Socioeconomic status

Special education status

Migrant status

Immigrant status

School type

In addition to the student-level variables, the propensity score model also included school

variables that added information about the school a student attended (to account for school-based

contextual differences, which may account for differences in the propensity for a student to

participate). A total of 123 variables were considered for the propensity score model. Data were

not available for each of these covariates for all students. To account for this, indicator variables

were used to model the relationship between the pattern of missing data and propensity to

participate in the program (Rosenbaum & Rubin, 1984). The propensity score model was fit

separately for each grade (Grades 3–12), and separately for each definition of treatment (30+

days; 60+ days). The final propensity score models for each grade were checked to ensure that

the analysis sample was balanced across relevant covariates. The propensity score models all

produced comparison samples that were balanced with the treatment across the 123 variables

examined for balance. This result indicates that the treatment and comparison groups had no

significant differences from one another (prior to treatment) as measured by these 123 variables.

Impact Analysis Results

Tables 33a and 33b show the effect of 21st CCLC programming on student reading and

mathematics achievement, cumulative GPA, percentage of credits earned, and number of

unexcused absences, pooled across grade levels (for both 30+ day and 60+ day treatment


definitions). It is important to note that the comparison group for the 30+ day and 60+ day

treatment definitions will differ. Separate propensity score models were fit for each, and it is

reasonable to think that students who attend 60 or more days are different from those who only

attend 30 or more days.

As shown in Table 33a, a statistically significant, positive impact of 21st CCLC was found for

reading achievement at the 0.01 significance level for both 30+ day and 60+ day treatments, with

students in treatment group achieving 0.027 standardized deviation units higher for the 30+ day

treatment and 0.033 standardized deviation units higher for the 60+ day treatment than students

in the comparison group. There was also a significant positive impact of 21st CCLC

programming on student mathematics achievement at the 0.01 significance level for both 30+

day and 60+ day treatments, with students in the treatment group achieving 0.044 standardized

deviation units higher for the 30+ day treatment and 0.035 standardized deviation units higher

for the 60+ day treatment than students in the comparison group. Although positive, these

program effects are quite small.

In terms of outcomes pertaining to high school students, there was a nonsignificant negative

impact of 21st CCLC on student cumulative GPA for the 30+ day treatment and significant

positive impact for the 60+ day treatment. For the 60+ day treatment, the cumulative GPA in the

treatment group was 0.195 standardized deviation units higher than that in the comparison group.

For percentage of credits earned, a significant positive impact was found at 60+ days of

participation. Regardless the significance of effect estimates all effect sizes are small (Cohen,

1988).

Table 33a. Impact of 21st CCLC on Achievement Pooled Across Grades

Subject Treatment Effect Size S.E.1 of Effect Size p

Reading2

30+ day 0.027 0.008 0.001

60+ day 0.033 0.011 0.004

Math3 30+ day 0.044 0.008 <0.001

60+ day 0.035 0.011 0.002

Cumulative GPA4

30+ day -0.022 0.026 0.399

60+ day 0.195 0.049 <0.001

Percent of credits earned4

30+ day 0.034 0.027 0.212

60+ day 0.144 0.048 0.003

1 Standard error

2 Include Grades 4–8, 10

3 Include Grades 4–8

4 Include Grades 9–12

In terms of unexcused absences, a statistically significant, negative effect of 21st CCLC was

found for the number of unexcused absences at the 0.01 significance level for both 30+ day and

60+ day treatments. In this regard, the number of unexcused absences was lower in the treatment

group than that in the comparison group at both 30 and 60 days of 21st CCLC. These effects

were moderate to large. More specifically, for the 30+ day treatment groups, the number of

unexcused absences in the treatment group was 66 percent of the level found in the comparison


group made up of nonparticipating students. For the 60+ day treatment group, the number of

unexcused absences in the treatment group was just 39 percent of the level found in the

comparison group.

Table 33b. Impact of 21st CCLC on Number of Unexcused Absences

Pooled Across Grades

Treatment Effect S.E. p Weighted Mean Ratio (Treatment/Comparison)1

30+ days -0.312 0.009 <0.001 0.657

60+ days

-0.638 0.017 <0.001 0.393

1Weighted Mean Ratio is the ratio of mean of unexcused absences after accounting the weight

generated for each student during PSM process.

Tables 34a and 34b show the impact on achievement and number of unexcused absences broken

down by grade for the 30+ day treatment group. In terms of reading achievement, there was a

statistically significant, positive impact of treatment on reading achievement for Grades 6 and 7,

and no significant impact for all other grades. There also was a significant positive impact of

treatment on mathematics achievement for Grades 6 and 8.

In terms of high school outcomes, a significant positive impact of 21st CCLC was found on

student cumulative GPA for Grade 11, and a significant negative impact was found for Grade 9.

Of some interest in Table 34a was that 21st CCLC participation did not have significant impact

on the percentage of credits student earned at a single grade, although as shown in Table 33a, a

significant impact was seen across all high school grades levels at 60 days of participation.


Table 34a. Impact of 21st CCLC on Achievement – 30+ Day Treatment

Grade Reading Math Cumulative GPA Percent of Credits Earned

Effect S.E. p Effect Size Effect S.E. p Effect Size Effect S.E. p Effect Size Effect S.E. p Effect Size

4 1.813 5.426 0.745 0.043 -0.240 10.727 0.983 -0.004

5 1.138 0.801 0.156 0.029 0.236 0.957 0.805 0.005

6 1.694 0.598 0.005 0.038 3.671 0.739 <0.001 0.066

7 1.459 0.768 0.057 0.031 1.338 0.933 0.152 0.022

8 0.842 0.880 0.339 0.016 3.833 0.909 <0.001 0.065

9 -0.189 0.047 <0.001 -0.186 -0.005 0.011 0.646 -0.024

10 -0.841 1.752 0.631 -0.016 0.006 0.043 0.895 0.006 0.013 0.010 0.189 0.060

11 0.099 0.045 0.028 0.121 0.014 0.011 0.196 0.071

12 0.039 0.053 0.470 0.052 0.003 0.012 0.834 0.016


In terms of unexcused absences, a statistically significant, negative impact was found for the

number of unexcused absences for all grades (3–12) at 30 days of participation (again, a negative

result is desired here because it indicates fewer unexcused absences). Table 34b indicates that the

number of unexcused absences in the treatment group was lower for all grades than in the

comparison group, with the largest impact happening in Grades 5–8.

Table 34b. Impact of 21st CCLC on Number of Unexcused Absences,

30+ Day Treatment

Grade Number of Unexcused Absences

Effect S.E. p Weighted Mean Ratio (Treatment/Comparison)

3 -0.376 0.037 <0.001 0.736

4 -0.387 0.041 <0.001 0.715

5 -0.550 0.042 <0.001 0.566

6 -0.459 0.028 <0.001 0.621

7 -0.394 0.027 <0.001 0.631

8 -0.444 0.028 <0.001 0.665

9 -0.185 0.032 <0.001 0.784

10 -0.350 0.021 <0.001 0.702

11 -0.092 0.024 <0.001 0.916

12 -0.095 0.029 0.001 0.853

Tables 35a and 35b are similar to Tables 34a and 34b, but they show the results for those

participating in 21st CCLC for 60+ days. A significant positive impact of 21st CCLC on reading

achievement was found for youth in Grade 6 (see Table 35a). In addition, there was a significant

positive impact of 21st CCLC on math achievement for youth in Grades 6 and 8 at the 60-day

threshold.

In terms of outcomes related to high school students, a significant positive impact was found on

student cumulative GPA for youth in Grades 10 and 11, and a significant negative impact was

found for youth in Grade 9. Also as shown in Table 35a, there was a significant positive impact

of 21st CCLC on the percentage of credits students earned for Grade 10 in particular. There are

some interesting findings here, particularly between Grades 9 and 10, which may warrant some

future investigation given the increasing effort being dedicated to ensuring students make an

effective transition to high school in Grade 9.


Table 35a. Impact of 21st CCLC on Achievement – 60+ Day Treatment

Grade Reading Math Cumulative GPA Percent of Credits Earned

Effect S.E. p Effect Size Effect S.E. p Effect Size Effect S.E. p Effect Size Effect S.E. p Effect Size

4 5.552 7.037 0.474 0.129 -3.785 9.492 0.710 -0.059

5 1.305 0.971 0.179 0.032 0.280 1.184 0.813 0.005

6 1.870 0.860 0.030 0.040 3.807 1.083 0.000 0.066

7 -0.103 1.305 0.937 -0.002 0.394 1.548 0.799 0.006

8 2.279 1.592 0.152 0.042 3.439 1.682 0.041 0.056

9 -0.192 0.093 0.040 -0.192 0.000 0.021 0.986 -0.002

10 4.074 2.865 0.155 0.077 0.278 0.073 0.000 0.278 0.066 0.016 <0.001 0.297

11 0.512 0.099 <0.001 0.652 0.022 0.020 0.281 0.117

12 0.135 0.102 0.186 0.189 -0.005 0.021 0.801 -0.031


As shown in Table 35b, a statistically significant, negative impact was found for the number of

unexcused absences for all grades (3–12) when treatment was defined as 60 days of participation

in 21st CCLC. Here again, the number of unexcused absences in treatment group was lower for

all grades than in the comparison group, with the largest impact happening in Grades 5–8.

Table 35b. Impact of 21st CCLC on Number of Unexcused Absences,

60+ Day Treatment

Grade Number of Unexcused Absences

Effect S.E. p Weighted Mean Ratio (Treatment/Comparison)

3 -0.482 0.045 <0.001 0.628

4 -0.478 0.055 <0.001 0.657

5 -0.805 0.060 <0.001 0.439

6 -0.947 0.054 <0.001 0.369

7 -0.950 0.056 <0.001 0.377

8 -0.653 0.053 <0.001 0.510

9 -0.161 0.062 0.010 0.794

10 -0.987 0.051 <0.001 0.420

11 -0.499 0.054 <0.001 0.503

12 -0.241 0.066 <0.001 0.691

Summary of Impact Analyses Results

Generally, findings from the impact analyses conducted in relation to youth outcomes associated

with the 2011–12 project period indicated positive program impacts across each of the outcomes

examined:

Significant, positive program impacts were found for both reading and mathematics at

both the 30-day and 60-day participation thresholds. Findings from the Year 1 report

included such effects for mathematics only and not for reading. However, effect sizes

were very small, ranging from .027 for reading at 30 days to .044 for mathematics at 30

days.

Significant, positive program impacts were found for both cumulative GPA and credits

earned / credits attempted at only the 60-day participation threshold. The effect size for

cumulative GPA at 60 days was .195, a small effect, and the effect size for credits earned

/ credits attempted at 60 days was .144, also a small effect.

Significant, positive program impacts were found in terms of a lower number of

unexcused absences at both the 30-day and 60-day participation threshold. The effect size

at 30 days was -.312, a moderate effect, and the effect size at 60 days was -.638, a large

effect.

It is important to note that the propensity score stratification approach employed here seeks to

minimize the impact of selection bias on the estimates of program impact. However, it is an


untestable assumption that such models can fully account for selection bias. To the extent that

other variables exist (not available for this analysis) that predict student participation in 21st

CCLC and are also related to student achievement or unexcused absences, these analyses may be

limited. To that end, these analyses provide initial evidence about the impact of 21st CCLC on

academic achievement and unexcused absences, but should not necessarily be considered

equivalent to experimental studies which have strong internal validity.


Conclusions

In conducting the 2011–12 statewide evaluation of the Washington 21st CCLC-funded

programming, a primary goal was to understand how and to what degree centers were

implementing research-supported practices in their programming, as well as what impact

participation in 21st CCLC-funded programming had on student outcomes such as reading and

math achievement, grade point average, and absences. Specifically, the evaluation questions that

guided the design are as follows:



2. To what extent was there evidence that centers funded by 21st CCLC implement

research-supported practices related to quality afterschool programming?








The 2011–12 evaluation built on the previous year by further developing the leading indicators

initiated in Year 1. The leading indicator system is designed provide grantees with data from the

evaluation to allow them to assess how they have adopted research-supported best practices,

what their strengths and weaknesses are, and how they might improve programming moving

forward. The leading indicators provide the structure for the correlational within-program

analysis, allowing for clustering of similar performance groups and examination of the

association between performance level on the indicators and youth outcomes.

In considering the relationship between the leading indicators and youth outcomes as examined

in Evaluation Question 3, it was hypothesized that centers scoring below average on key

indicators would be negatively associated with youth outcomes. This hypothesis was supported

for centers in the Instructional Context-Content Below Average cluster, which includes indicators

such as intentionality in program design, linkages to the school day, and use of student data to

inform programming. In this case, there was a negative association between centers with low

scores on these indicators and student performance on state reading assessments and a positive

association with unexcused absences. Similarly, membership in the Organizational Context

Below Average cluster, which includes training, program climate, and reported internal

communication, was negatively associated with youth attendance in the 21st CCLC program.

However, this analysis also revealed findings that did not support this hypothesis. Membership in

the Organizational Context Below Average cluster was positively associated with cumulative

GPA and credits earned, and membership in the Instructional Context-Process Below Average

cluster was positively associated with higher levels of program attendance, although this may be

due to a relatively large number of centers serving elementary students only in this cluster, as


elementary school students tend to have higher levels of program attendance than middle or high

school students.

In addition to membership in below-average clusters for the leading indicators, various student-

and center-level characteristics were examined with relation to student outcomes. Analysis

revealed that a positive relationship existed between the number of days of participation in 21st

CCLC programming and academic performance outcomes, and a negative association existed

with unexcused absences from school. In addition, centers with school-based status were

negatively associated with youth performance on math and reading assessments and positively

associated with unexcused absences. Several student characteristics were also found to have a

negative association with academic performance, including eligibility for free or reduced-price

lunch, receipt of special education services, limited English proficiency, and minority status.

With regards to the final evaluation question, the evaluation also examined how program

participation impacted youth outcomes using a propensity score matching approach to reduce

selection bias. Exploring the impact of participation in the 21st CCLC programming by

comparing participants with nonparticipants revealed positive program impacts when pooled

across all grades, although not necessarily for youth at all grade levels. The analysis revealed

small but significant impacts on both reading and math achievement when pooled across grades,

whereas the findings for Year 1 revealed positive impacts for math only.

There were also significant but small positive effects of the 21st CCLC program on cumulative

GPA and percentage of credits earned for students in the 60-day treatment group, and students in

both the 30-day and 60-day treatment groups showed significantly lower levels of unexcused

absences than nonparticipants. In the case of program effects on unexcused absences, program

effects were moderate to large. This analysis provides a basis for continued exploration of the

impact of 21st CCLC programming on participants, especially in divergent impacts on individual

grade levels.

This report’s findings on leading indicators, correlational relationships, and impact analyses

provide guidance for grantees on areas for continued growth in the upcoming years, including (1)

using data to inform services for individual students, (2) allowing staff more time for planning

and preparation, and (3) identifying ways to incorporate more youth ownership into the program

at grade-appropriate levels. These results are very similar to those identified in the Year 1 report.

In addition, there appears to be some evidence that (a) there are opportunities for growth in terms

of how centers go about designing and delivering activities from a content perspective and (b)

that enhanced levels of practice in this area are related to better school-related outcomes.

Although OSPI has an infrastructure for supporting instructional quality from a process

perspective, it may want to give consideration to the types of supports it could provide to

enhance the manner in which 21st CCLC supports the cultivation of skills and knowledge from a

content perspective, particularly in relation to the needs of participating youth.

Although a variety of positive program effects were demonstrated in this year, OSPI is interested

in further exploring the types of impacts 21st CCLC is having on social-emotional learning, 21st

century skills and competencies, and noncognitive outcomes. Toward this end, in Year 3 of the

evaluation, AIR will be working to collect information from grantee project directors on what


they believe their programs are impacting in these areas and what their priorities should be in

terms of testing measurement strategies to assess program impact on such outcomes. Steps will

be taken to select a sample of instruments designed to measure high-priority outcomes and pilot

those in a small number of centers during the spring semester of the 2013–14 school year.

Finally, the leading indicators represent a substantial investment of time and effort to provide

Washington 21st CCLC grantees with actionable data to guide and support program

improvement efforts. A key goal of the Year 3 evaluation will be to better understand the

efficacy of these tools as a vehicle for supporting quality improvement efforts and to highlight

portions of the system that are proven to have especially high value to grantees and OSPI.


References

Birmingham, J., Pechman, E. M., Russell, C. A., & Mielke, M. (2005). Shared features of high-

performing after-school programs: A follow-up to the TASC evaluation. Washington DC:

Policy Studies Associates.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (second edition).

Hillsdale, NJ: Lawrence Erlbaum Associates.

Durlak, J. A., & Weissberg R. P. (2007) The impact of after-school programs that promote

personal and social skills. Chicago: Collaborative for Academic, Social, and Emotional

Learning.

Eccles, J., & Gootman, J. A. (Eds.). (2002). Community programs to promote youth

development. Washington, DC: National Academies Press.

Glisson, C. (2007). Assessing and changing organizational culture and climate for effective

services. Research on Social Work Practice, 17.

Granger, R. C., Durlak, J. A., Yohalem, N., & Reisner, E. (2007). Improving after-school

program quality. New York: William T. Grant Foundation.

Granger, R. (2008). After-school programs and academics: Implications for policy, practice, and

research. Social Policy Report, 22(2), 3–19. Ann Arbor, MI: Society for Research in

Child Development. Retrieved from

http://www.srcd.org/documents/publications/spr/spr22-2.pdf

Hong, J., & Hong, Y. (2009). Reading instruction time and heterogeneous grouping in

kindergarten: An application of marginal mean weighting through stratification.

Educational Evaluation and Policy Analysis, 31(1), 54–81.

Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. A., Snow, D., & Martin-Glenn, M.

(2006). Out-of-school-time programs: A meta-analysis of effects for at-risk students.

Review of Educational Research, 76(2), 275–313.

Little, P. (2007). The quality of school-age child care in after-school settings (Research to Policy

Connections, No. 7). New York: National Center for Children in Poverty, Columbia

University Mailman School of Public Health.

Naftzger, N., Bonney, C., Donahue, T., Hutchinson, C., Margolin, J., & Vinson, M. (2007). 21st

Century Community Learning Centers (21st CCLC) analytic support for evaluation and

program monitoring: An overview of the 21st CCLC performance data: 2005–06.

Naperville, IL: Learning Point Associates. Retrieved from

http://www2.ed.gov/programs/21stcclc/2501yr3ppics.doc

Rosenbaum, P. R., & Rubin, D. B. (1984). Reducing bias in observational studies using

subclassification on the propensity score. Journal of the American Statistical Association,

79(387).

http://www.srcd.org/documents/publications/spr/spr22-2.pdf

http://www2.ed.gov/programs/21stcclc/2501yr3ppics.doc


Smith, C. (2007, March). Predictors of quality at the point of service provision: Empirical and

methodological background for the YPQA field trial. Presented at the biennial meeting of

the Society for Research in Child Development, Boston.

Vandell, D. L., Reisner, E. R., Brown, B. B., Dadisman, K., Pierce, K. M., Lee, D., et al. (2005).

The study of promising after-school programs: Examination of intermediate outcomes in

Year 2. Madison, WI: Wisconsin Center for Education Research. Retrieved from

http://childcare.wceruw.org/pdf/pp/year2_executive_summary_and_brief_report.pdf

Wilson-Ahlstrom, A., & Yohalem, N. (with Pittman, K.). (2007). Building quality improvement

systems: Lessons from three emerging efforts in the youth-serving sector. Washington,

DC: Forum for Youth Investment, Impact Strategies.

Yohalem, N., Wilson-Ahlstrom, A., Fischer, S., & Shinn, M. (2009). Measuring youth program

quality: A guide to assessment tools (2nd ed.). Washington, DC: Forum for Youth

Investment.

Zief, S. G., Lauver, S., & Maynard, R. A. (2006). The impacts of after-school programs on

student outcomes. Campbell Systematic Review, 3, 1–52. Retrieved from

http://www.campbellcollaboration.org/lib/download/58/

http://childcare.wceruw.org/pdf/pp/year2_executive_summary_and_brief_report.pdf

http://www.campbellcollaboration.org/lib/download/58/

American Institutes for Research Washington 21st CCLC Program Evaluation: Year 1

LOCATIONS

Domestic

Washington, D.C.

Atlanta, GA

Baltimore, MD

Chapel Hill, NC

Chicago, IL

Columbus, OH

Frederick, MD

Honolulu, HI

Indianapolis, IN

Naperville, IL

New York, NY

Portland, OR

Sacramento, CA

San Mateo, CA

Silver Spring, MD

Waltham, MA

International

Egypt

Honduras

Ivory Coast

Kenya

Liberia

Malawi

Pakistan

South Africa

Zambia

ABOUT AMERICAN INSTITUTES FOR RESEARCH

Established in 1946, with headquarters in Washington, D.C.,

American Institutes for Research (AIR) is an independent,

nonpartisan, not-for-profit organization that conducts behavioral

and social science research and delivers technical assistance

both domestically and internationally. As one of the largest

behavioral and social science research organizations in the world,

AIR is committed to empowering communities and institutions with

innovative solutions to the most critical challenges in education,

health, workforce, and international development.

14-1167_01/14

20 North Wacker Drive, Suite 1231

Chicago, IL 60606-2901

312.288.7600 | Fax: 312.288.7601

www.air.org

Documents

Washington 21st Century Community Learning Centers Program