Policy Studies Associates

After-School Activity Observation Instrument (AOI)

Observation Manual

Adapted for the WCER/PSA

Study of Promising After-School Programs

Do not distribute or use without permission of the authors

Developed by Ellen Pechman and Carolyn Marzke Policy Studies Associates, Inc.

2003 Edition

Acknowledgements

The AOI was developed to standardize observations of after-school program practices for evaluation and research studies. The instrument’s conceptual framework and its indicators build on documented evidence of best practices used in high quality child-care and after-school programs, and draws on observation instruments and quality standards from several sources. These resources are listed in the bibliography. Several colleagues contributed substantially to the development and testing of the AOI. We especially appreciate the assistance of Jennifer Birmingham of PSA and Beth Miller of NIOST, who helped to conceptualize and refine the instrument’s design, rating categories, and indicators. We also gratefully acknowledge the assistance of the following colleagues who piloted and informed further refinements to the instrument and observation procedures: Megan Welsh, Christina Russell, Dwayne Smith, and Lara Fabiano, at Policy Studies Associates; and Kim Dadisman, Sarina Meyer, Kay Lukszys, and Kim Pierce at the Wisconsin Center for Education Research, University of Wisconsin-Madison.

2

Table of Contents

1. Overview of the After-School Activity Observation Instrument 2. Conducting the Observations

Selecting Activities to Observe Domains for Observation Sample Selection Sample Observation Matrix Observing Activities

3. Program Overview Data Collection Bibliography Appendices

A – After-School Activity Observation Instrument (AOI) B – After-School Activity Observation Instrument: Definitions

3

1. Overview of the After-School Activity Observation Instrument

This manual provides the instructions and instruments for the observational component of assessments of the quality of after-school programs in conjunction with the Wisconsin Center for Education Research/Policy Studies Associates’ Study of Promising After-School Programs. It is designed to guide researchers who will be conducting on-site visits using the After-School Activity Observation Instrument (AOI; Appendix A), for observing and coding process quality features during after-school program activities.

The goal of the AOI is to collect eight 15-minute observations over a two-day period of time.

These observations of the program in action are structured to cover the range of program activities that youth in the target grades typically experience over a two-day period.

The AOI uses the study’s theory of change to structure and systematically record site visitors’ observations of the after-school program in action. The instrument has two components. The first part, Activity Observation Coding, enables researchers to record data about the types of activities offered and the presence or absence of important indicators of program quality within observed activities. The instrument’s second part, Program Overview and Summary, is used at the conclusion of all activity observations to record descriptive information about program facilities and activity schedules and to summarize the overall quality of the physical environment, literacy environment, and organization of the program.

Activity Observation Coding: In this part of the instrument, observers maintain a record of youth

activities and their interactions with staff and with peers. Observations occur for 15 minutes at a time in a particular activity space, during which observers (1) record the types of activities and activity context, and (2) the quality of the interactions, opportunities, and resources available to youth. Two forms—each with several components—guide data collection in these areas:

1. Activity Context Coding Form: This is the cover sheet for each 15-minute observation

segment. It captures information about the content and skill areas of observed activities; types of space and materials used; numbers of adults and youth participants; grade levels; and grouping patterns and strategies. Most of it can be easily completed at the start of the observation, although some details may be confirmed and revised as the observation concludes.

2. Activity Observation Coding Form: This form captures information about the quality of interactions, opportunities, and resources in three five-minute time intervals (or observation segments). The observer marks indicators observed during each five-minute observation segment, according to the instructions and definitions that are provided in Appendix B. There are six categories of observation indicators:

• Youth Interactions: positive and negative peer interactions among youth

• Staff-Youth Interactions: positive and negative interactions among staff and

youth

• Youth Engagement: degree of youth task orientation and focus

• Opportunities for Skill-Building and Mastery: strategies staff use in teaching, coaching, and facilitating youth learning and mastery; ways in which youth are engaged to promote learning and mastery of cognitive and non-cognitive skills

4

• Activity Organization: indicators of how well activities are planned and set up

• Setting and Resources: indicators of the availability of appropriate materials,

space, and other resources to support the activity Program Overview and Summary: This part of the instrument provides a place for observers to

describe certain aspects of the program as a whole, including: the various facilities/space(s) the project uses, the types and durations of activities offered during the course of a week, the overall physical environment, the overall literacy environment, and program organization. Only one form is completed per site visit. It represents a consensus perspective of all the observers who visit the site during each site visit period, based on the observations of program activities and information obtained from the program director. The AOI Program Overview should be supplemented by a brief, several-paragraph vignette that tells a compelling story or illustrates a particularly instructive interaction among youth and staff. This vignette should illustrate a dimension of the “best program practice” the researcher observed while on site. The vignette should be brief, pithy, and compelling. It should include one or more illustrative quotes from staff and/or students so that, as much as possible, staff and students’ voices are conveyed and the story clearly demonstrates why it was selected as an exemplar. 2. Conducting the Observations Selecting Activities to Observe Determining how many observations to conduct. The goal is to collect at least eight 15-minute observations during the two-day site visit. If there are two observers, this translates into four observations per observer. If possible, observers should observe on different days, so that on each afternoon one member of the team is observing while the other is free to conduct interviews or other meetings. If there is one observer, this translates into four observations per afternoon over the course of two afternoons. Dividing the observations over two days ensures that visitors see the widest possible range of activities in sites whose schedules vary from day to day. To the extent possible, site visitors should observe a sample of program activities over the entire period of their observation. If two visitors are on site, this will make observing eight activities relatively easy over a two-day period. It may be more of a challenge if there is only one site visitor, but observing at least eight separate activities is very important to providing a true optimal picture of the range of opportunities typically available to youth. Visitors should use their judgment about how to balance their observation time across the two days with other site visit tasks. Decisions about how to balance observations across the two visit days should be guided by the goal of visiting the widest range and most representative sample of activities in which children and youth participate regularly. Emphasis should be on typical activities—those that are held on a weekly or daily basis. If special events occur that involve students in the targeted grades, they can be observed, but only for one of the available 15-minute segments.

Organizing observations on site. To ensure that the broadest scope of activities is observed,

each day site visitors should select one activity from each of the four domains listed below. In consultation with the site leaders, site visitors should select these activities from the project schedule, being mindful to observe across the grade levels of interest (third/fourth and/or sixth/seventh) as well as activities. Site visitors should verify with the program coordinator that each scheduled activity will be

5

occurring in the location stated. A sample observation matrix is provided below. Please note that while snack can be an important part of an after-school project, observations should be geared toward more substantive activities. Domains for Observation Sample Selection

1. Academic/cognitive - Homework time - Academic enrichment (content-based)/literacy development - Debate teams

2. Arts

- Recreational arts and crafts - Visual arts enrichment/instruction - Performing (incl. dance) enrichment/instruction

3. Sports/Fitness

- Open drop-in sports/movement/physical games - Structured athletics/teams

4. Other

(e.g., service clubs/activities, career exploration) If possible, try to schedule observations of both:

1. Staff-led instruction/practices/classes in a particular content area (e.g., cooking class, film-making class or project, basketball team practice, dance class).

2. Unstructured recreational opportunities supervised but not led by staff or other adults

(e.g., open gym, table/board games, open time for socializing/hanging out, computer time) Sample Observation Matrix

Day 1 Activity Observed Younger Group Older Group

Academic/cognitive Math enrichment Homework time

Arts Recreational arts and crafts Dance rehearsal

Fitness Open/gym/dodge ball Sports team practice

Other Cooking activity Board games

6

Day 2 Activity Observed Younger Group Older Group

Academic/cognitive Homework time Chess club

Arts Mural painting project Step rehearsal

Fitness Tumbling class Martial arts class

Other Arrival/snack Arrival/snack

Observing Activities

Getting Settled. Enter the room or activity space as unobtrusively as possible, finding a place to sit where you will be out of the way, but can still observe staff and participants. If it is not too disruptive, introduce yourself to the staff member.

Completing the Activity Context Coding Sheet. As soon as possible, record the time the observation began in the space provided and begin observing/coding the Activity Context Coding Sheet. Some may be codable immediately, while other indicators on this sheet may be checked at the conclusion of the observation, or at any time that the information is clear. Some pointers on completing this sheet:

1. Activity numbers should be assigned by each observed in the order of their observations (the

first observation of that site would be assigned Activity Number 1, the second observation would be assigned Activity Number 2, etc.)

2. Use the Site IDs designated for your program site. 3. Check all relevant activity types. 4. If you observe a transition check “other” activity type and write “transition” on the back of

the form.

5. If you observe a homework activity, circulate to determine which subjects students are working on. Make sure each subject is indicated in the skill areas section of the form.

6. For the total adults/staff and number of youth sections of the form, write the number of

people in the space provided approximately the middle of the second 5-minute observation segment.

Completing the Activity Observation Coding Sheet. When you have completed the Activity

Context Coding Sheet, begin the observation using the Activity Observation Coding Sheet and a notepad. Each observation consists of three 5-minute scoring segments. Use a stop-watch to keep track of your observation time. Observe for 4 minutes and record your observations on the data collection form in the final minute. Repeat this strategy for the next two observation segments.

If multiple activities are occurring within the activity space, try to pay attention to each. Be sure

to circulate if necessary to be able to hear staff-youth interactions. Keep the following procedures in mind as you observe and record:

7

¬ Check an indicator only if it describes a “predominant” condition during the five-minute

observation. A condition or behavior is predominant if:

⇒ It occurs more than once or persists during the five-minute observation segment.

⇒ When there are multiple groups of children and/or multiple staff:

• Predominance for peer interactions, youth engagement, and youth skill-building opportunities means that the behavior occurs in more than one group of youth and/or it is so pronounced and persistent within a specific group of youth that it contributes to the overall social/emotional climate and/or learning environment. One youth’s behavior within one group may be a large factor in certain settings. On the other hand, in most groups there are always fleeting conflicts or infractions that pass unnoticed and do not detract from the overall environment or activity.

• Predominance for staff-youth interactions and staff skill-building strategies is

assessed from the perspective of staff behavior rather than the number of youth/groups of youth with whom staff interact. An indicator assessing staff behavior is predominant if staff consistently demonstrate the behavior over the course of the observation segment, regardless of how many youth they interact with during that period. If staff are highly engaged with one group of youth, they can be rated highly engaged, even though they have no interactions with other groups of youth in the room. If in the course of this focused interaction staff ignore or are unaware of pressing needs in other groups—for assistance, intervention in conflicts, etc.—then the indicator describing this lack of awareness would also be marked.

When is an indicator “predominant?”

• It occurs more than once or it persists during the five-minute observation segment among one or more individuals or groups

• It is so pronounced and persistent that it contributes to the overall

social/emotional climate and/or learning environment

• “Predominance” of youth behaviors is assessed from the point of view of the targeted group of youth

• “Predominance” of staff behaviors toward youth is assessed from the

point of view of staff (regardless of the number of youth affected)

¬ If specific examples of behavior or actions either by youth or staff were particularly interesting, describe them briefly on an accompanying pad of paper, recording direct quotes to the degree possible. (These notes may be used at the conclusion of the observation in writing up a vignette that reflects what was observed.) Note date and activity number in your notes.

8

¬ Maintain a list of questions that arose as you scored, note date and activity number, and

confirm scoring before leaving the site by reviewing questions with co-visitor.

Completing the Activity Observation. Once you have concluded the observation, mark the time the observation was completed in the in the space provided. You should have completed one Activity Context Coding Form and one Activity Observation Coding Sheet with three observation segments per activity. As you leave the setting, if it is not too disruptive, try to thank the staff member for allowing you to visit and move on to the next activity on your list. 3. Program Overview Data Collection

There are two components of the final program overview data collection: a descriptive summary

of program facilities and activity schedules, and an assessment of several cross-cutting elements of program quality. Only one form should be submitted for each site visit. Once the observations are complete, the site visitors should collaborate to complete both sections of this data collection form. The lead site visitor (the person who is responsible for scheduling and confirming the visit) is responsible for leading discussions among site visitors needed to complete the form, completing the form, and circulating it for review/revision among visitors.

A few tips on completing the Program Overview Data Collection Form:

1. Record the number of rooms the program uses by type of room, and describe what activities occur in that room or how it is used. Indicate which rooms were observed on the site visit by checking the “observed” column.

2. List the activities that occurred on the days of observation within each activity

category. List each activity, and next to each activity, indicate the age or grade of youth who participated in that specific activity as well as the scheduled duration of the activity in minutes; check whether the activity was observed.

3. For each indicator of the quality of the physical environment, literacy environment,

and program organization, decide with the other observers whether the indicator is “very true,” “somewhat true,” or “not true” for that project. Provide descriptions of the project that support your rating in the space allowed.

9

Bibliography American Youth Policy Forum. (1997, 1999). Some things do make a difference for youth: A

compendium of evaluations of youth programs and practices. Washington, DC: American Youth Policy Forum.

Catalano, R.F., Berglund, M.L., Ryan, J.A.M., Lonczak, H.S., & Hawkins, J.D. (1998). Positive youth

development in the United States: Research findings on evaluation of positive youth development programs. Seattle, WA: University of Washington, Social Development Research Group.

Eccles, J., & Gootman, J.A. (Eds.). (2002). Community programs to promote youth development.

Washington, DC: National Academy Press. Grossman, J. B., Price, M. L., Fellerath, V., Jocovy, L. Z., Kotloff, L.J., Raley, R., and Walker, K.E.

(2002). Multiple choices after school: findings from the extended-service schools initiative. Philadelphia, PA: Public/Private Ventures.

Harms, T., Jacobs, E.V., White, D.R. (1996). School-Age Care Environment Rating Scale. New York:

Teachers College Press, Columbia University. National School-Age Care Alliance. The NSACA Standards for Quality School-Age Care. Boston, MA:

Author. National School-Age Care Alliance and the National Institute for Out-of-School Time (1998). Guide to

NSACA Program Accreditation: NSACA Program Observation Tool. Boston, MA: NSACA. Policy Studies Associates, Inc. (2001). School Literacy Observation Instrument. Developed for The

Qualitative Evaluation of the Achievement First Initiative, Fund for Educational Excellence, Baltimore, Maryland.

Policy Studies Associates, Inc. (2002). The TASC After-School Classroom Observation Instrument.

Developed for the Evaluation of the TASC After-School Program. Washington, DC: Policy Studies Associates, Inc.

Rosenthal, R., & Vandell, D.L. (1996). Quality of care at school-aged child care programs: Regulatable

features, observed experiences, child perspectives, and parent perspectives. Child Development, 67, 2434-2445.

The Safe and Sound Campaign (1999). Standards for Baltimore After-School Opportunities in Youth

Places. Baltimore, MD: Author. Scales, P.C., & Leffert, N. (1999). Developmental assets: A synthesis of the scientific research on

adolescent development. Minneapolis, MN: Search Institute. Smith, L.J., & Ross, S.M. (1999). School Observation Measure Observer’s Manual. Memphis, TN:

Center for Research in Educational Policy, The University of Memphis. Vandell, D.L., & Pierce, K.M. (2001, April). Experiences in after-school programs and child well-being.

In J.L. Mahoney (Chair), Protective aspects of after-school activities: Processes and mechanisms. Paper symposium conducted at the biennial meeting of the Society for Research in Child Development, Minneapolis, MN.

10

Vandell, D.L., & Posner, J.K. (1999). Conceptualization and measurement of children’s after-school environments. In S.L. Friedman & T.D. Wachs (Eds.), Assessment of the environment across the lifespan. Washington, DC: American Psychological Association.

Walker, K. E., Grossman, J. B., & Raley, R. (2000). Extended service schools: putting programming in

place. Philadelphia, PA: Public/Private Ventures.

11

Appendix A

After-School Activity Observation Instrument (AOI)

12

13

Activity Context Coding

Site I.D.: Date: Room #: Activity #: Start: End: Observer Initials:

Description of Observed Activity:

A. FOCUS C. KNOWLEDGE/SKILL AREAS F. TOTAL ADULTS/STAFF #

1. Single activity/whole group 1. Interpersonal 1. Number of adults

2. Primary focal activity/opt-out activity(ies)

2. Physical/athletic

2. Number of teens

3. Multiple activities/no focal activity 3. Artistic

B. ACTIVITY TYPE 4. Math/numeracy

1. Homework assistance 5. Reading/language arts G. GRADE LEVELS

2. Tutoring 6. Problem-solving/

decision making

K 1 2

3. Study skills/test preparation

7. Other (Specify on back)

3 4 5

4. Reading/language arts enrichment 8. None 6 7 8

5. Math/science enrichment D. TYPE OF SPACE Other:

6. Recreational reading/ listening to story/book

1. Classroom/activity room

H. NUMBER OF YOUTH #

7. Computer skill-building 2. Gym 1. Total number of youth

8. Computer games 3. Auditorium 2. Number of boys

9. Sports: competitive and non-competitive games; tournaments

4. Cafeteria

3. Number of girls

10. Sports: practice/drills/skill-building

5. Library I. GROUPING PATTERNS

(Check all that apply)

11. Fitness/exercise class (including martial arts)

6. Computer room

1. Large Group (15+)

12. Arts and/or crafts 7. Outside playground/

field

2. Medium (6-14)

13. Performing arts rehearsal/ instruction (including dance, music, and drama)

8. Other (Specify on back)

3. Small groups or pairs (2-5)

14. Board/table/card games or puzzles

E. MATERIALS USED 4. Individuals working

alone

15. TV/video watching 1. Computers

16. Higher education or career orientation 2. Art supplies

17. Health/well-being 3. Sports equipment

18. Service/civic (in community or program)

4. Reading materials

19. Snack 5. Writing materials

20. Arrival/dismissal 6. Musical instruments

21. Cultural awareness clubs/projects 7. Games

22. Other (Specify on back) 8. Other (Specify on back)

14

Activity Context Other/Not Applicable Descriptions

Activity Type

Skill Areas

Type of Space

Materials Used

15

Activity Observation Coding

Site ID Date Room #: Activity # Start Time: End Time: Observer Initials:

J. YOUTH INTERACTIONS

Segment Segment Indicator Checklist (+) YOUTH… A B C

Indicator Checklist (-) YOUTH… A B C

1. Have few or no interactions

2. Interactions are positive 6. Have disagreement/disputes that are unresolved

3. Cooperate/share with each other 7. Harass/intimidate/threaten each other

4. Talk informally/listen to each other

5. Appear to enjoy each other’s company

K. STAFF-YOUTH INTERACTIONS

Segment Segment Indicator Checklist (+) STAFF… A B C

Indicator Checklist (-) STAFF… A B C

1. Engage positively with youth 8. Have few or no interactions with youth 2. Are highly engaged with youth (check both

this category and the one above) 9. Use arbitrary or harsh disciplinary methods

3. Engage informally in conversations with youth

10. Belittle youth or embarrass them in front of peers

4. Use positive language and tone of voice with youth

11. Appear to be unaware of teasing/bullying or other serious conflicts among youth

5. Listen actively and attentively to youth 12. Fail to encourage/facilitate participation from disengaged youth

6. Use positive behavior management techniques OR no behavior issues observed

7. Praise/encourage individual youth

YOUTH… YOUTH…

13. Respond to staff directions 16. Ignore directions/tune staff out

14. Interact positively with staff 17. Are rude/actively negative toward staff 15. Seek out positive contact/interactions with staff

L. YOUTH ENGAGEMENT

Segment Segment Indicator Checklist (+) YOUTH … A B C

Indicator Checklist (-) YOUTH… A B C

1. Are on-task/actively participating 3. Are off-task/distracted/floating without purpose

2. Are interested/focused 4. Are disruptive to the activity/prevent peers from focusing

5. Number of youth engaged/on-task (end of segment)

6. Number of youth present (end of segment)

16

Site ID: Date: Room #: Activity #: Start Time: End Time: Observer Initials:

M. OPPORTUNITIES FOR SKILL-BUILDING AND MASTERY

Segment Segment Indicator Checklist (+) STAFF… A B C

Indicator Checklist (-) STAFF… A B C

1. Are clearly focused on instruction/helping youth to learn something new or master a skill

22. Criticize youths’ work without offering guidance

2. Communicate goals, purpose, or expectations for activity

23. Give answers/impose solutions without encouraging youth to solve own problems

3. Provide direct instruction/lecture/give directions

24. Provide poor answers or wrong answers to questions

4. Demonstrate or model a concept or skill 25. Fail to respond in a timely manner to youth who ask for assistance

5. Ask youth “why”, “how”, and “what if” questions that require complex answers

26. Do not notice youth who are having difficulty

6. Actively facilitate discussion among youth 7. Constructively critique/offer feedback to individual youth

8. Challenge youth to push themselves intellectually, creatively and/or physically

YOUTH …

9. Actively listen to/watch a lecture/presentation/ performance

10. Write (not for homework) 11. Read or are read to (not for homework)

12. Use math (not for homework) 13. Work cooperatively with each other to

achieve a goal

14. Work on projects with culminating products or events

15. Choose what or how they do something or help determine the direction of an activity

16. Lead activities, individuals, or groups of peers

17. Tutor/mentor other youth 18. Think strategically/analyze/solve complex problems

19. Use additional information to accomplish a goal/make a decision

20. Participate in structured discussions that contribute to completing the activity/task

21. Discuss interpersonal needs/feelings with peers and/or adults

17

Site ID: Date: Room #: Activity #: Start Time: End Time: Observer Initials:

N. ACTIVITY ORGANIZATION

Segment Segment Indicator Checklist (+) A B C

Indicator Checklist (-) A B C

1. Youth know what they are expected to do 4. Youth do not know what is expected of them

2. The work area is appropriately prepared and ready for youth

5. Staff are unprepared for the activity

3. Activity is appropriate for youths’ skill levels 6. Multiple staff work at cross purposes

O. SETTING AND RESOURCES

Segment Segment Indicator Checklist (+) A B C

Indicator Checklist (-) A B C

1. If materials or equipment are needed, they are functional/in reasonably good condition, OR no special materials are required.

4. Materials/equipment/space is needed but not available or inadequate for the activity/task

2. If materials are needed, there are enough for all youth to have the access they need to participate, OR no special materials are required.

5. Space is unsafe for the activity/task

3. There is enough room for the activity/number of youth

6. External interruptions/distractions disrupt activity/prevent staff and/or youth from focusing

18

AOI Program Overview Data Collection Form

Program ID Code: Date: Observe ID Code:

P. Types of Available Space # # Observed Comments 1. Classroom/activity room

2. Gym

3. Auditorium

4. Cafeteria

5. Library

6. Computer room

7. Outside playground/field

Facility Description

List and describe the space(s)

available to the program and how

they are used

8. Other

19

Q. Activities Offered Grades Req’d? Yes = 1 No = 0

# Days/ WEEK

# Mins/ Day

Obsvd Yes = 1 No = 0

1. Homework/tutoring

Activity #1

Activity #2

Activity #3

2. Cognitive/academic/enrichment activities (Subject area or inter-disciplinary learning activities that are NOT homework or tutoring)

Activity #1

Activity #2

Activity #3

3. Artistic activities (Theater, dance, photography, painting, crafts, etc.)

Activity #1

Activity #2

Activity #3

4. Physical recreation activities (Including formal and informal sports/recreation, martial arts, fitness, cheerleading, etc.)

Activity #1

Activity #2

Activity #3

5. Community/family-linked activities (Including on- or off-site community service activities, mentoring, etc.)

Activity #1

Activity #2

Activity #3

6. Open time/general recreation

Activity #1

Activity #2

Activity #3

7. Other

Activity #1

Activity #2

Activity Schedule

By program area, list each specific activity offered during the WEEK that you are on site. For each activity listed, indicate ALL the grades served, number of days PER WEEK it is offered, duration of activity in minutes, and write a 1 or 0 to indicate if the activity was observed.

Activity #3

20

Overall Environment and Organization

Indicators On the days observed… Very

True Somewhat

True Not

True

Brief Description

R. Physical Environment

1. There is comfortable furniture available for relaxing, reading, etc.

3 2 1

2. There is quiet space/time for youth to read or do quiet work 3 2 1

3. The program has access to a play area that is sufficiently large to allow youth vigorous physical play/exercise

3 2 1

4. There are no observable safety hazards 3 2 1

S. Literacy Environment

1. Age-appropriate reading materials (books, magazines) are readily accessible to youth

3 2 1

2. There are enough reading materials for youth 3 2 1

3. There are printed materials on the walls 3 2 1

T. Overall Program Organization

1. Activities generally happen as scheduled 3 2 1

2. In general, youth are directed, focused, and involved (e.g., not out of control)

3 2 1

3. Arrival, dismissal, and transitions are reasonably smooth

3 2 1

Appendix B

After-School Activity Observation Instrument: Definitions

21

Activity Context Coding Definitions

FOCUS

ACTIVITY TYPE

Homework help Youth work on homework assignments. If youth participate in academic activities that are not homework, do not mark this category.

Test preparation/study skills Instruction and activities designed expressly to prepare students for standardized achievement tests or to teach them study skills. Activities may include practicing different types of test questions, or taking practice tests.

Tutoring Youth work individually or in small groups with a tutor, who helps them work on a particular skill such as reading, math, or English language skills.

Reading/language arts enrichment

Math/science enrichment

Enriched supplementary instructional content in either reading/language arts or math/science that goes beyond homework, develops broad-based conceptual/cognitive learning. It may or may not be directly related to school content/assignments, but is not a homework assignment.

Recreational reading/listening to story/book Reading or listening to a story or book not assigned for homework; reading to accomplish a goal (e.g. functional literacy skills, reading a play for pleasure/practice, etc.)

Computer skill-building This category refers to activities involving learning to use the computer, NOT to the use of the computer to complete home work, to build academic skills, or to search the web (unless instruction focuses on teaching youth to use the web).

Computer games Youth play games on the computer. They are not learning to use the computer, or using the computer to complete homework, practice test-taking skills, practice academic skills, or for an academic enrichment activity.

Sports—competitive or non-competitive games; tournaments Supervised or non-supervised games using athletic skill, indoors or outdoors.

Sports—practice/drills/skill-building Preparation and training in a sport.

Fitness/exercise class (including martial arts) Activities targeting general physical fitness/strength, including aerobics, martial arts, weight lifting, yoga.

Arts and/or crafts Visual arts-based activities (painting, drawing, sculpture, photography, videography) and crafts (knitting, pottery).

Performing arts rehearsal/instruction (including dance/music/drama) Lessons, practices/rehearsals, recreational performances or performance arts that are for the purpose of self-expression. This area includes hip-hop, step, drill team. It does NOT include fitness-oriented types of movement.

22

Board/table/card games or puzzles These may include games of luck, strategy games, jigsaw puzzles, puzzle games; board games (e.g., chess, Monopoly), table games (e.g., pool, air hockey), card games (e.g., Hearts, Crazy 8s).

TV/video watching Youth watch TV or videos, either for recreation or as part of some other activity. If the TV/video is incorporated into another activity, mark the type of activity that provides the context (e.g., watching a performance as part of a dance class).

Higher education or career orientation Activities that provide information about and/or opportunities to plan for higher education or jobs/careers.

Health/well-being Health education, conflict resolution, life skills education activities.

Service/civic (in community or program) Planning or assisting with projects that support the quality of community life or foster program-school or community-school linkages.

Snack Eating or preparing for snack.

Arrival/dismissal Youth are arriving/gathering at the beginning of the program for attendance; youth are gathering for dismissal/departure.

Cultural awareness clubs/projects Activities/projects that develop cultural, religious, or ethnic awareness, understanding, or identity.

Other Describe here content areas/activities observed that cannot be categorized or grouped above.

KNOWLEDGE/SKILL AREAS: Mark these knowledge/skill areas when instruction/skill development are intentional. This can include presentations/lectures on a substantive topic or practice of specific skills, but does not include informal “pick-up” activities initiated by youth for

“fun.” These areas should be marked only when there is evidence that the knowledge/skill development is focused and intentional. Either students or staff may initiate/lead the activity.

Interpersonal Negotiation, conflict resolution, and/or other communication skills. Do not mark for purely social interactions.

Physical/athletic Physical skills related to sports, fitness, or physical games (including martial arts, yoga, step, cheerleading, gymnastics, etc.)

Artistic Artistic skills, working in any medium (visual, musical, dance, dramatic, photographic, video, etc.)

Math/numeracy Mathematics learning, computational skill development/practice

Reading/language arts Reading, writing, literacy skill development/practice

Problem-solving/decision making Developing skills in making practical decisions or solving practical or conceptual problems. E.g., life-skills decision making around healthy choices; or planning a project or performance (deciding music, parts, actions).

23

Other Check this category AND DESCRIBE here specific other types of skills not listed above (e.g., research, critical thinking, ).

None Check if no discernable skill-building is occurring during the activity. Describe what is happening to support this conclusion.

Classroom Gym

Auditorium

Cafeteria Library Computer lab Outside playground/field

Type of space

Other

Check the one location in this category that best applies to the setting in which the activity takes place. If “other” is marked, specify the type of space used.

Computers

Art supplies

Sports equipment Reading materials Writing materials Musical instruments Games

Materials used

Check all the types of technology used during each observation segment. More than one box may be checked in each observation.

Adults Total adults/staff

Teen staff

Count and RECORD in the space on the right the NUMBER of adults and the number (if any) of teen staff in the room.

Total Number of Youth (girls and boys) Indicate the total number of youth, the number of girls, and the number of boys observed. If in doubt about gender, make an educated guess rather than disrupting the class. Check tally of boys and girls against total number of youth.

Grade Level Observed (Check all that apply)

Circle all the grades that are represented in the classroom or activity space. More than one category can be checked, if appropriate. You may use prior knowledge about what grade levels are supposed to be in the room if it is difficult to determine on sight, or check with staff later and circle based on their response.

24

Large group (15+)

Medium group(s) (6-14)

Small group(s) or pairs (2-5)

Grouping Patterns

Individuals working alone

This category describes the grouping clusters that occur during the observation segments. Check all the category(ies) that BEST describe the size of groups observed, e.g., if a medium group, and two sets of pairs are working together, check both the “medium: and “small groups or pairs” categories. More than one category can be checked.

25

Activity Observation Coding: Indicator Definitions

1. YOUTH INTERACTIONS

Youth… Definition

Have infrequent interactions with one another Youth work on their own; interactions are intermittent and rare. Only a few youth interact with one another, or youth interact with one another only with a brief question or offering of assistance. Mostly youth are working independently without conversation.

Interactions are positive Tones of voice and/or body language are friendly and positive.

Cooperate/share with each other Youth share materials and space, help/accept help from each other, jointly work out how to approach an activity or situation.

Talk informally/listen to each other Youth have conversations that are either social or task-focused.

Appear to enjoy each other’s company Youth seek each other out; they seem to like each other, want to be around one another, and/or have fun together.

Have disagreements/disputes that are unresolved Youth pick on one another, use “put-downs,” and/or belittle one another. Disputes, harsh words, or anger persist during the observation but do not lead to serious intimidation or threats.

Harass/intimidate/threaten each other Youths’ petty arguments or disagreements become threatening or intimidating and go beyond teasing or belittling to threats of physical harm.

26

2. STAFF-YOUTH INTERACTIONS

STAFF…

Definition

Engage positively with youth Staff have occasional interactions with youth which are generally positive, or pay attention to youth sporadically. These interactions are constructive and generally supportive, but they are not continuous or substantive.

Are highly engaged with youth Staff actively and continuously interact with youth in positive, substantive ways, engaging them, talking with them, and/or participating in activities with them. They pay attention to and show interest in what youth are doing and/or how they are working together.

Engage informally in conversations with youth Staff initiate informal conversations with youth (in contrast to task-oriented or instructionally focused conversations). They respond to youths’ efforts to talk with them by showing interest and extending the conversation.

Use positive language and tone of voice with youth Staff use affirming words and tone of voice, speaking in a manner that indicates respect, appreciation, and belief in the value and potential of youth.

Listen actively and attentively to youth Staff look at youth when they are speaking and respond to youths’ comments and questions, affirming what was said, asking questions for clarification, or responding directly to a request or question.

Behavior management issues do not arise, or staff use positive management techniques to resolve behavior problems.

Staff set appropriate limits and communicate expectations for behavior. They intervene constructively and calmly to address disruptive behavior, redirecting youth away from disruptive or negative behavior and explaining/discussing why the behavior was unacceptable. Check this category when no behavior management issues arise or when staff manage them constructively.

Praise/encourage individual youth Staff praise youths’ efforts, accomplishments, and progress, and offer encouragement to youth who are frustrated.

YOUTH…

Respond to staff directions Youth follow the directions of staff in a positive, agreeable way.

Interact positively with staff Youths’ interactions/conversations with staff are friendly and respectful. Tones of voice and body language are positive.

Seek out positive contact/interactions with staff Youth initiate dialogue; offer to help staff; ask their opinions; banter back and forth.

Ignore directions/tune staff out Youth deliberately ignore staff who are talking to them. They may pretend to listen but are

not listening, or they simply ignore staff directions or comments.

Are rude/actively negative toward staff Youth are actively negative toward staff, showing hostility, disrespect, or intentional efforts to offend staff.

27

2. STAFF-YOUTH INTERACTIONS (CONTINUED)

STAFF…

Definition

Have few or no interactions with youth Staff do not interact with youth. They may appear bored or distant rather than actively interested in what youth are doing or how they are doing.

Use arbitrary or harsh disciplinary methods Staff yell at, shame, or disparage youth in response to negative or disruptive behavior. Staff punish the whole group for infractions by one youth.

Belittle or embarrass youth in front of peers Staff correct youth publicly in a way that embarrasses them, or intentionally belittle or embarrass youth when they make mistakes

Appear to be unaware of teasing/bullying or other serious conflicts among youth

Staff are not present when these occur, are present but appear unaware that they are happening, or appear to be ignoring them.

Fail to encourage/facilitate participation from disengaged youth

Staff do not acknowledge youth who are bored, wandering, or inactive. They fail to encourage participation or provide alternatives for those who are disinterested or uninvolved.

3. YOUTH ENGAGEMENT

YOUTH…

Definition

Are on-task/actively participating Youth are engaged in the focal activity(ies) and/or are using free time appropriately.

Are interested/focused Youth appear to be interested in the activity, seem to find it absorbing. Their participation seems to extend beyond compliance, to be motivated by their own interest as much or more than the fact that it is required.

Are off-task/distracted/floating without purpose Individuals or groups of youth are not participating in the activity (either actively or passively watching) and/or have difficulty settling into an activity during free time. Youth are wandering or are engaged in inappropriate conversations. Mark this category if, for the bulk of the segment, more than one youth is disengaged and clearly without purpose.

Are disruptive to the activity/prevent peers from focusing Youth are exceedingly disruptive and staff are unable to refocus/redirect them productively. The disruption may be caused by a small number of youth, or even one youth, but affects the entire group.

Number of youth “on task” at end of each observation segment The number of youth actively engaged in the activity at the end of each 5-minute observation segment.

Number of youth present at end of each observation segment The number of youth present in the room at the end of each 5-minute observation segment.

28

4. OPPORTUNITIES FOR SKILL-BUILDING AND MASTERY

Staff…

Definition

Are clearly focused on instruction/helping youth to learn something new or master a skill

Through staff’s actions and words, it is clear that they are trying to impart knowledge or teach a particular concept or skill to youth.

Communicate goals, purpose, or expectations for activity

This is an outcome-oriented (not process) expectation. Staff explain why youth are participating in the activity and/or what they expect youth to accomplish. Do not mark here to record when staff state rules for behavior or re-state what youth are allowed to do in a particular activity (these are behavior management strategies).

Provide direct instruction/lecture/give directions

Staff are telling, focusing, directing, with the goal of teaching, reviewing information, or explaining to youth how to do something. This interaction can include a series of simple question-and-answer exchanges with youth involving brief factual answers.

Demonstrate or model a concept or skill

Staff show and explain the specific steps/processes to follow in completing activity/tasks. The demonstration/explanation goes beyond giving simple instructions to actually showing how to do something (by performing a dance step, demonstrating a drawing technique, working through a problem, etc.) or giving youth examples in response to questions.

Ask youth “why”, “how”, and “what if” questions Staff ask youth questions that require extended responses (i.e., sentences or paragraphs rather than one- or two-word or short-phrase answers). They ask youth to explain their answers/arguments, respond to counterarguments, or to attempt to answer their own questions.

Actively facilitate discussion among youth Staff encourage dialogue between youth through discussion prompts or structured opportunities for group discussions among youth. They encourage a dialogue among youth rather than question/answer or call/response statements. The dialogue may be task-related or informal or personal.

Critique/offer constructive feedback to individual youth

Staff circulate and actively review/observe youths’ work/activities, and offer specific suggestions to youth for improving their skills or performance. They use positive words and tone of voice; offer concrete, supportive advice; and provide youth with examples of what they like about their work/progress.

Challenge youth to push themselves intellectually, creatively and/or physically

Staff communicate high expectations and encourage youth to “push themselves” and expand their skills/experiences by moving to the next level of difficulty.

29

4. OPPORTUNITIES FOR SKILL-BUILDING AND MASTERY

STAFF…

Definition

Criticize youths’ work without offering guidance Staff express general dissatisfaction with youths’ work without helping youth to understand how they might improve.

Give answers/impose solutions without encouraging youth to solve own problems

Staff design/impose solutions to interpeersonal problems/conflicts rather than helping youth to develop their own solutions/compromises. Staff respond to youth questions with direct answers or tell them what to say (how to answer a question), rather than respond to youth with questions or prompts designed to help youth figure out the answer for themselves.

Provide poor answers or wrong answers to questions Staff give incomplete, confusing, or incorrect answers to youth questions. For example, in providing assistance with homework/academic enrichment, they provide information that is misleading or wrong.

Fail to respond in a timely manner to youth who ask for assistance

When youth ask for help, staff do not or are unable to provide assistance in time to allow youth to remain focused and progress through the activity.

Do not notice youth who are having difficulty (and who need assistance)

Staff seem unaware of how youth are doing, do not notice when youth are having difficulty (even if youth do not speak up) and offer assistance.

30

4. OPPORTUNITIES FOR SKILL-BUILDING AND MASTERY (CONTINUED)

YOUTH…

Definition

Actively listen to/watch lectures/presentations/ performances

Youth are the audience, watching and/or listening and paying close attention to a lecture, presentation, or performance by an adult or a peer. May also include watching videos chosen for instructional (rather than purely entertainment) purposes.

Write (not for homework) Youth are engaged in a writing a story, poem, or piece of nonfiction that is not a homework assignment. They write independently of a worksheet and are not writing in response to a series of questions, but may respond to a writing prompt.

Read or are read to (not for homework) Youth are reading a book, magazine, poem or newspaper that is not a homework assignment or are being read to by someone else.

Use math skills (not for homework) Youth are learning or practicing math skills as part of an activity that is not a homework assignment.

Work cooperatively with each other to achieve a goal Youth work together toward a common goal, negotiating roles, dividing tasks, and helping each other as needed.

Work on projects with culminating products/events The activity youth are engaged in builds to a product or event designed to showcase their work/skills—an art piece or art show, theatrical production, service project, sale, trip, etc.

Choose what or how they do something or help to determine the direction of the activity

Youth get to make choices about what they do or the approach they take to completing a task. Youth have opportunities to make suggestions about how to carry out activities, tasks, or events that occur during the observation.

Lead activities, individuals, or groups of peers Youth lead some part of the activity by organizing a task or a whole activity, or leading a group of youth within the activity.

Tutor/mentor other youth Youth are either formally assigned as tutors or mentors to other youth in the activity, or they informally reach out to provide substantial help to other youth who are struggling with the activity.

Think strategically/analyze/solve complex problems The activity requires that youth develop a plan or think ahead several steps and anticipate others’ responses (e.g., game of chess, tic-tac-toe, budgeting for a trip or a project) or solve a complex problem that requires substantial thought.

Use information to accomplish a goal or to make a decision Youth must gather and/or synthesize information in order to complete a task or make a decision. The information-gathering and processing should have a purpose, defined either by staff or chosen by youth. Examples include internet searches to help design a project or plan a trip.

Participate in structured discussions that contribute to completing the activity/task

Youth are discussing a topic among themselves or with an adult. The discussion must clearly be an intentional part of the activity.

Discuss interpersonal needs/feelings with peers and/or adults

Youth have discussions with peers and/or adults that are purposeful and intentional efforts to express feelings and resolve interpersonal problems constructively. May be formal/structured and involve the whole group, or they may arise in the course of resolving interpersonal conflict or addressing disruptive behavior.

31

5. ACTIVITY ORGANIZATION

Definition

Youth know what they are expected to do Youth seem to know what to do; if observed, the instructions staff give youth are easy to understand and to follow.

The work area is appropriately prepared and ready for youth

The area or setting in which activities occur are ready when youth arrive. If they are needed, materials they are readily accessible and/or efficiently dispersed to youth.

Activity is appropriate for youths’ skill levels The activity is not so difficult that most youth have trouble participating successfully and become frustrated, nor is it so easy that most youth master it quickly and become bored. Activity accommodates multiple age/skill levels if present, allowing youth to participate successfully at own pace.

Youth do not know what is expected of them Youth seem confused or unsure about what to do; if observed, the instructions staff give youth are confusing, unclear, or difficult to follow.

Staff are unprepared for the activity Staff have not prepared the room or assembled needed materials. Staff seem unprepared to lead the activity, are unsure about how to direct youth.

Multiple staff work at cross purposes If there is more than one staff member in the room, they do not support one another, give conflicting information or directions, or otherwise do not work well together.

6. SETTING AND RESOURCES

Definition

If materials or equipment are needed, they are functional/in reasonably good condition, or no special materials are required.

Materials or equipment, if needed, are in a condition that allows them to be used as they were intended. Sports equipment is functional, games have all their pieces, computers are working and relatively up-to-date, etc. Mark positively if no materials are needed or used.

If materials are needed, there are enough for all youth to have the access they need to the materials they need to participate, or no special materials are required.

There are enough materials to allow all youth present to participate in the activity simultaneously. Youth are not effectively excluded from the activity because there are too few materials or pieces of equipment. Youth can make progress on a task or activity with the materials available. Mark positively if no materials used needed or used.

There is enough room for the activity/number of youth

The space accommodates the number of youth and/or activities under way without undue disruption or crowding that prevents youth from enjoying the activity or making progress on a particular task.

Materials/equipment/space is needed but not available or is inadequate for the activity/task

Lack of materials or equipment or space constraints prevent youth from participating successfully in the activity.

Space is unsafe for the activity Clear safety-hazards exist that makes the space inappropriate for use in an after-school program or the space is unsafe for the activity at hand (e.g., there is broken glass all over the room or there is nothing wrong with the

room, but it is far too small/cluttered with furniture to be used for the football activity you observe).

External interruptions/distractions disrupt activity/prevent staff and/or youth from focusing

The space is arranged/shared in such a way that external disruptions are clearly distracting to staff and/or youth and disruptive to the activity.

32