1

Quality Matters:Building Capacity and Investment

in Youth Program Quality

The Center for Youth Program Quality

2

Objectives

• Understand how quality defined and measured by the Youth Program Quality Assessment and other quality assessment tools

• Learn about quality improvement systems currently implemented in numerous statewide and place-based networks

3

Systems for Quality

Leveraging existing “change” resources

POS QualityAccess to key

developmental and learning experiences

Professional Development

Local Evaluators

AccountabilityMeasurement

4

At best, schools fill only a portion of developmental “white space.”

Who fills the rest? And what is the “locally appropriate mechanism for

monitoring the availability, accessibility, and quality of

programs…” in school and out?

Saturating communities with “ample programs” requires

improving the quality and reach of all the systems, settings and

programs that touch young people’s lives. Selective

replication and improvement are not sufficient.

schoolAges

Times of Day/Year

Outcome Areas

after-school

Developmental “White Space”

Quality & Reach… Looking into the developmental white space

5

Ch

ildren

’s Services in

LA

Co

un

ty

SOURCE: Margaret Dunkle

The Landscape of Youth Programs

6

Capacity to Recruit, Train,

Retain Workforce

Cross-system convening/coordination mechanisms

Widely adopted assessment and monitoring proceduresBuy in re definitions, quality standards, accountability requirements

Strong Policy / Leadership Horsepower

Decision-maker engagement & coordinating structures Aligned policies for quality accountability and improvement

Shared vision, strong demand, active family/youth involvement

Capacity to Assess & Improve

Programs

Professional development opportunities/incentives

Adequate assessment and improvement training/TA capacity

Strong, Stable Program Base

Healthy program landscape (distribution & focus)

Accurate data on workforce (skills, supports, recruitment, retention)Cross-system provider networks and communications

Domains Elements

Cross-system program data base/info source

The Ready by 21™ Quality Counts Framework

7

What is program quality?

???inputs

inputs

inputs

outcomes

youth program

Another way to say it:•What do we want to see in high quality youth programs?

Defining Quality

Several ways to organize:• Readin’ + ‘Ritin + ‘Rithmatic (old-school)• Affect + Active Learning + Metacognition (Education)• Relatedness + Autonomy + Competence (Psychology SDT)

Content Therapeutic process

What kids need…Relationship + Task + Increasing Complexity

What adults should do…

9

Our Quality Construct: The Pyramid of Program Quality

PlanMake choices

Reflect

Partner with adults

Lead and mentorBe in small groups

Experience belonging

Engagement

Reframing conflictEncouragementSkill building

Active engagementSession flow

Welcoming atmosphereSupportive

Environment

Interaction

Safe Environment Healthy food and drinks

Program space and furniture Emergency proceduresPsychological and emotional safety

Physically safe environment

10

Viewing Quality in a Systems Context

SAESystem Accountability

Environment

PLCProfessional

Learning Community

POSPoint Of Service

Safety

Support

Interaction

Engage

11

Defining Quality What can management do?

Learning Features

CommunityContinuity

PerformanceFeedback

Data & Information

External monitoring

Lesson & curriculum review

Daily supervision & support by site coordinator

Pre-session planning

Frequent staff meetings

ManagementPriorities/Values

Youth engagement & content relevance

Purposeful relationships

Youth voice structures

FocusingFeatures

12

Engagement

Interaction

Supportive Environment

Safe Environment

Professional Learning Community

6 standards

4 standards

33 standards

19 standards

1 standard

Other 32 standards

49 standards

Youth Organizational Voice

Source: (1998).The NSACA Standards for Quality School-Age Care. There are thirty-six keys of quality and 144 total standards

Defining quality at the system levelCurrent Regulatory Models Miss the POS

13

The Youth Program Quality Assessment (YPQA)

– Validated observational assessment tool

– Measures quality at the point of service

– Assesses frequency and access to key developmental experiences

– Can be used to assess progress over time

– Content-neutral for use across settings, ages, systems

14

Form BOrganizational InterviewAsk questions, write, score

(2 hours)

Form AObservation

Watch, write, score (3 hours)

Organization

Program Offering 1

Program Offering 2

Program Offering 3

Program Offering 4

Structure of the Youth PQA

15

Sample Item from the Youth PQA

III. InteractionIII-L. Youth have opportunities to develop a sense of belonging.Note: Structured refers to the quality of being intentional, planned, and/or named; it does not refer to informal conversation.

Indicators Supporting Evidence

1 Youth have no opportunities to get to know each other (beyond self-selected pairs or small cliques).

3 Youth have informal opportunities to get to know each other (e.g., youth engage in informal conversations before, during, or after session.

5 Youth have structured opportunities to get to know each other (e.g., there are team-building activities, introductions, personal updates, welcomes of new group members, icebreakers, and a variety of groupings for activities)

The Youth PQA consists of7 Domains (4 in A, 3 in B)30 Scales (18 in A, 12 in B); 103 item rows (60 in A, 43 in B)

“Domain”“scale”

“item row”

What POS Quality Looks Like on the Ground

Occurred For All

• Sample of nearly 600 different youth workers

• Parallel findings in schools research

Occurred For Some

Did Not Occur

8/13/2008 16The Center for Youth Program Quality

Does it Work? Findings from Several Samples

• POS quality-outcomes findings:– Supportive environment related to: Attendance– Interaction related to: Interest in program– Engagement related to: Sense of challenge, sense of growth,

school-day reading, school-day suspension– Note: No offerings get to high engagement without high support

and high interaction

• Quality Improvement (YPQI) Findings– Scores increase from pre to post– Scores increase in the targeted areas more– Management practices are related to quality change (Vision,

Feedback, Continuity)

How we think about DDCI- People change not programs

Contemplation

Preparation

Action

Maintenance

Quality assessment

Planning with data

Implementation & coaching

Repeat cycle

SAESystem Accountability

Environment

PLCProfessional

Learning Community

POSPoint Of Service

Individual Change Model

Organizational Context

Prochaska, J.O., & DiClemente, C.C. (1982). Transtheoretical therapy toward a more integrative model of change. Psychotherapy: Theory, Research and Practice, 19(3), 276-287.

8/13/2008 18The Center for Youth Program Quality

Competencies for Youth Development Practitioners

Emergence of POS quality culture Spend time planning & prepping activities

Improve practice systematically

"Team focused on youth experience" mentality

 

Intentionality in POS quality practiceIcebreakers and inclusion

Cooperative grouping strategies

Targeting learning edge – scaffolding, engaging youth in planning & reflection, active learning

Providing opportunities for youth voice and leadership

 19

Competencies for Management

– Knowledge/understanding of positive youth development methods individually and integrated as the High/Scope participatory learning approach;

– Conduct reliable performance assessment based on observation and validated measurement rubrics;

– Conduct performance coaching based on strengths based transmission of performance data;

– Lead a staff team through a data driven quality improvement process

20

System Level Outputs

– Accountability policy with high returns in staff buy-in and learning;

– Common language and terminology supporting focus on quality at the point of service;

– An integrated professional development investment that is integrated over time as a sequence of PD experiences, integrated across levels of organization and professional roles, and integrated with ultimate program purposes in positive youth development;

21

Outcomes

–High quality programs

–Better staff retention and retention of better staff

–Greater impact on youth development and learning

22

23

mbus

etroit

Minneapolis

`` `

l

Kentucky

Iowa

Oklahoma

New York

Rhode Island

Austin

Georgetown Divide Columbus

Indianapolis

Grand Rapids

Nashville

St. Louis

Washington*

West Palm Beach County

Rochester

Chicago

Systems for QualityAccountability Policies in Places

• YPQA is part of state and county accountability policies:–Cross sector (DHS& DOE) snapshots: Iowa, Washington, Arkansas–Statewide 21st Century: Michigan, Maine, Minnesota, Rhode Island, New Mexico, –Cities and Counties: Rochester, Detroit, Grand Rapids, Palm Beach

24

Designing Quality Improvement Systems (QIS)

System Capacity

Program Staff Skill

& Knowl

Self-assessment of Quality (b)

Planning with Data (c)

External quality assessment (a)

Quality Advisor (e)

TOTs for quality assess, coaching,

and youth work methods (f,g)

Coaching & Training

Phase 1: Readiness & Capacity

Phase 2: Impact & Sustainability

Targeted youth work methods

training for direct staff (h)

Quality coaching by managers (i)

Use of on-line dashboards and

training (d)

External Quality Report with Norms

Self-Assessed Quality Report

Use of on-line dashboards and

training (d)

Mostly Mangers Managers and direct staff

25

Defining the Purpose of Your QIS

Lower Stakes

ProgramSelf-Assessment

Rough data toget staff thinking anddiscussing programquality in the context

of best practice

Less timeLess money

Impact internalaudiences

Higher Stakes

ExternalAssessment

Precise data forinternal and external

audiences for evaluation,monitoring, accountability,

improvement, reporting

More timeMore money

Impact internal andexternal audiences

(the creative middle)

Columbus IndianaPhase 1: Building Local Capacity

STEP 1

Decide to build system

STEP 2a

Self-assessment

STEP 2b

External assessment

STEP 3

Plan for improvement

STEP 4

Carry out plan

SAESystem

Accountability Environment

PLCProfessional

Learning Community

August 25Quality Matters Presentation

August 26Youth PQA Basics

January 27-28Ext Assessment

October 8Planning with DataImprovement Plan

Opt Phase 2Method WorkshopsQuality Coaching

STEP 5

Measure change

AnnuallyProgram SAExt AssessmentObserve-ReflectionPlanning with Data

POSPoint

Of Service

Questions about…

Purpose?

Process?

Pilot?

Next Steps…27