Upload
bonnie-johnston
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
Quality in AfterschoolApril 28, 2015
Agenda
1. Welcome 2. Context setting3. Research4. Believe It – Build It: MN’s Guide to Effective
Practices in Afterschool 5. Quality Assessment and Process Examples6. Funder Table Talks 7. Funder Table Talk Report Back, Commitments
and Questions
Context Setting
Sheila OehrleinMinnesota Department of Education
Research on Quality
Deborah Moore & Kari Denissen CunnienUniversity of Minnesota & Ignite Afterschool
Measuring Youth Program Quality
Facilitator: Deborah Moore, Director – Youth Work Learning Lab
The Youth Work Learning Lab Presents…
Data Paralysis
The Youth Work Learning Lab - School Of Social Work
The gauntlet of evaluating in youth programs
Hard for youth workers to measure and collect (time, resources and expertise)
Can’t help determine changes needed in program or by staff
Hard to use many tools across diverse program environments
Difficult for programs to evaluate causal information
Different funding sources require different measures
Using poorly designed tools
Tools are too expensive
Tools don’t help staff be more clear about what to “do”
Not enough learning and training support
Other accountability “burdens” never reduced – keep piling on The Youth Work Learning Lab -
School Of Social Work
I start with these ideas
Data should help us understand something we do not know, but it is better if it relates to our practical experience
Data should not be a full time job for youth programs
Data should be easy to use for change
Data is only one way of understanding the world, others matter too…
The Youth Work Learning Lab - School Of Social Work
Quality Measures Are Not Your Typical Outcome
Based on solid, consistent and compiling research.
Looks at settings and/or behaviors vs. youth internal states (i.e. harder to measure youth outcomes).
Based on things where we have higher levels of control (our practices).
Based where we have the most commonality across the diverse ways we do our work (how we work with youth, not content).
The Youth Work Learning Lab - School Of Social Work
Researchers Agree on Program Quality Elements
The National Research Council & Institute for Medicine list the following key features for quality programs:
Physical and psychological safety Appropriate structure Supportive relationships Opportunities to belong Positive social norms Support for efficacy and mattering Opportunities for skill-building Integration of family, school and community
efforts
- Community Programs to Promote Youth Development, 2002
Research Says Quality Matters
• Empirical evidence linking quality with positive impact
(Durlak, 2007; Vandell, 2005; 2007; 2009; 2011;
Sambrano, 2005 Catalano, 2005, Eccles & Gootman, 2002; Roth, Brooks-Gunn, Murray & Foster, 1998).
• Negative consequences of poor quality programs
(U.S. Dept. of Health, 2006; Belle, 1999).
• If we assess program quality, we can improve it. (Smith, 2012).
What does high quality look like?
Creating the space for young people to make decisions and have a choice about things that matter to them.
http://www.youtube.com/watch?v=Z2Z17g-2rIc
The Youth Work Learning Lab - School Of Social Work
BELIEVE IT. BUILD IT.
Kari Denissen CunnienIgnite Afterschool
We Believe Statements
Building Blocks
Continuous Program Improvement Cycle
Intentional Program
Design
Supportive Relationships &Environment
Youth Voice & Leadership
Community & Family
Engagement
Responsiveness to Culture & Identity
Organizational Management, Staff Support & Youth Safety
Safe & Active, Focused & Explicit
Are these effects “meaningful”?• Aspirin on heart disease, d = .03• Early child care & math achievement, d = .09-.12• Class size reductions on math achievement, d = .23Durlak, J.A., & Weissberg, R.P. (2010). American Journal of Community PsychologyAbove charts modified from a presentation by Deborah Vandell
A community approach
Gambone, M.A., Klem, A.M. & Connell, J.P. (2002). Finding Out What Matters for Youth: Testing Key Links in a Community Action Framework for Youth Development. Philadelphia: Youth Development Strategies, Inc., and Institute for Research and Reform in Education.
Human Development Approach
“the quality of a human life involves multiple elements whose relationship to one another needs close study….one of the appealing features…is its complexity: it appears well equipped to respond to the complexities of human life and human striving.”**
*Measure of America, A Project of the Social Science Research Council. Retrieved from: http://www.measureofamerica.org/human-development/**Nussbaum, Martha C. (2011) Creating Capabilities: The Human Development Approach.
Human development is defined as the process of enlarging people’s freedoms and opportunities and improving their well-being. Central to the human development approach is the concept of capabilities. Capabilities—what people can do and what they can become— are the equipment one has to pursue a life of value.*
Continuous Program Improvement
Youth Program Quality Assessment (YPQA)
Deborah MooreUniversity of Minnesota
David P. Weikart Center for Youth Program Quality · QUALITY COACHING
The YPQA Measures Differently
1. A validated instrument designed to assess the quality of youth programs at the point of service (POS--where youth and adults interact) and identify volunteer/staff training needs.
2. A set of items that measures youth access to key developmental experiences.
3. A tool which produces scores that can be used for comparison and assessment of progress over time.
4. Identifies staff training needs and encourages improvement and creation of a professional learning community
Higher scores are associated with higher levels of youth engagement
Youth self-reports of: o Belongingo Interesto Challenge o Learning
YPQA practices connects to social and emotional skills…
o Efficacyo Communicationo Empathyo Problem Solving
Youth Program Quality Assessment (Measures)
Pick a card and have a chat about why that practice matters to young people…
Top 5 Things Funders Should Know About Using the YPQA
o Observational assessment is scary AND affirming for youth workers, tread with care.
o The YPQA is useful as a self-assessment process where staff observe their own program or when external assessors observe (but these are not the same).
o Assessment is the Beginning – for the data to have power, it must be placed in the context of continuous quality improvement (emphasis on improvement) and data must get in the hands of those that work directly with youth.
o There are advantages for using one tool (creates common language across staff and programs, allows for community to aggregate data, focuses shared training on mutual improvement goals across programs, etc.)…but some choice can also be good.
o Quality assessment works BEST if it is about learning and improvement. Funders help when they focus on encouraging participation and improvement.
o It takes time- to change habits, to build organizational capacityThe Youth Work Learning Lab -
School Of Social Work
More Info or Ideas?
To download the YPQA free of charge go to:
Weikart Center for Youth Program Quality- Download the YouthPQA or http://www.cypq.org/downloadpqa
For info on classes and consulting using the YPQA go to:
Deborah Moore - Director Youth Work Learning LabUniversity of MN – Peters Hall1404 GortnerSt. Paul, MN 55108612-625-7813 [email protected]
Academic Skills PQA
Kara Bixby & Nora RobinsonSaint Paul Public Schools Foundation
31 2014 Saint Paul Public Schools Foundation
Youth Program ScenarioExtended day learning class for 6th
graders.Conducting an architecture project
to promote learning of STEM concepts.
Objective is to learn about angles and proportions.
Group discussion: What will you have the class do to learn these skills in a high-quality way?
32 2014 Saint Paul Public Schools Foundation
Key QuestionsHow do we help youth programs
provide high-quality academic opportunities?
What do they look like in out-of-school time?
What practices are most important?
Defining Academic Skill-Building
Academic Support
Intentionality
Academic Skill-
Building
=LEARNIN
G
34 2014 Saint Paul Public Schools Foundation
Overview of the ToolMeasures the quality of academic
skill-building opportunities.Expansion of skill-building scale
in Youth Program Quality Assessment (YPQA).
Alignment with Youth Program Quality Intervention (YPQI).
Research-based and validated.Assists in identifying areas where
staff practice can be improved.
35 2014 Saint Paul Public Schools Foundation
Settings for UsePrograms that provide academic
supports for youth21st Century Community Learning
CentersSchoolsExtended day learning opportunitiesCamps and museum programsCollective impact models with
academic goals
36 2014 Saint Paul Public Schools Foundation
Why Use?
Geared toward program improvement and performance management.
Focused on practices shown to be effective for academic skill-building.
Offers a higher standard of performance.
Helps programs show the research-based practices they are using.
37 2014 Saint Paul Public Schools Foundation
Why Use? (cont.)
Includes items specific to social and emotional learning.
Focused on skill development.Additional point of information
and data for academics.Prepares young people to benefit
from school/core instruction.
38 2014 Saint Paul Public Schools Foundation
Additional Information
Download tool at Weikart Center website:
http://www.cypq.org/downloadpqa
Twin Cities info session to be scheduled Fall 2015
39 2014 Saint Paul Public Schools Foundation
Questions?
Kara BixbyResearch & Evaluation Director
651-325-4210
Nora RobinsonTutoring Partnership Director
651-325-4244
Quality Mentoring Assessment Path (QMAP)
Mai-Anh KapankeMentoring Partnership of Minnesota
Quality Mentoring Assessment
Path (QMAP)
2007 Conceptual work with a Quality Advisory Committee (QAC)
2008-’09 Tool Development and QAC beta testers
2009-’10 36 programs piloted - officially launched October 2010
2013 QMAP expansion - Connecticut, Nebraska and Washington
2014 QMAP expansion - Maryland
2015: Ohio enters agreement to use QMAP
MILESTONES…
QMAP is: Continuous improvement Built around a cycle Accommodates varying
program models Guided by quality
assurance goals Confidential
QMAP is not: Accreditation process One-time evaluation Used with start-up
programs A way to rank or score
programs Public reporting system
WHAT IS QMAP?
Divided into three sections Program Design & Management Program Operations Program Evaluation
QMAP ONLINE TOOL
Links to free resources for every item
Includes research and templates
RESOURCES
2 - 4 hours Develop priorities for
Improvement & Innovation Plan with program staff, not just a score or ratings
Plan becomes the platform for technical assistance with individual programs
ON-SITE REVIEW
Summary Packet sent: Cover letter Final Improvement &
Innovation Plan Statement of
Accuracy QMAP Benefits
Overview
FOLLOW-UP
Dedicated Partners
Complete the QMAP process
Expert Partners
100% of outlined best practices in place
QUALITY PARTNER RECOGNITION
Smaller programs are honored to receive the same level of recognition as their peers from larger programs
It’s a great way to share program information with staff, Boards and top leadership of multi-service organizations
QMAP helps programs leverage funding
QMAP builds the field toward quality
When programs have best practices in place, our families, kids and volunteers benefit
“MESSAGES FROM THE FIELD”
Expanded Learning Solution Plan
Jaimee BohningNorthside Achievement Zone
NAZ SEAL OF EFFECTIVENESS
• Partners following evidence-based Solution Plans
• Track real-time results for programs and individuals
• Blueprint for replication
Collective learning and performance accountability
• Monthly convening
• Includes senior NAZ staff, Board members, Parent Advisory Board members, and rotation of partner leaders
• Analysis to find technical and adaptive solutions
• Building culture of accountability: partners are accountable to each other and to families
RESULTS NAZROUNDTABLE ANALYSIS OF DATA
HIGH TOUCH, HIGH TECH PARTNERSHIP
EXPANDED LEARNING SOLUTION PLAN
• AGREED UPON RESULT:“Expanded Learning enrolled scholars demonstrate accelerated progress to achieve grade-level performance in math and reading.”
EXPANDED LEARNING SOLUTION PLAN
Solution Plan “buckets”• Organization Commitment• Reading Skills • Math Skills• Supporting Scholar Engagement• Transition Preparation and Support• Continuous Quality Improvement• Partnerships with Schools and Community Organizations• Family Engagement
EXPANDED LEARNING SOLUTION PLAN
• We aim to answer three key questions:– How much?– How well?– Is anybody better off?
What is working and how can we do more of it?Focus on scalability and replication
What isn’t working?Why not?Is it still important? If so, how do we improve?If not, how do we cease resource allocation?
Table Talk
1.RFP Application Process 2.Site Visits3.Communicating quality to your Stakeholders
Table Talk Report Back
1.What were key highlights from each table?
2.What will you take away from today and bring back to your organization?