View
213
Download
0
Category
Preview:
Citation preview
Perspectives on Impact Evaluation
Cairo, Egypt March 29 – April 2, 2009
Presented by:
Wayne M. Harding. Ed.M., Ph.D. , Director of Projects, Social Science Research & Evaluation, Inc., Burlington, MA USA wharding@ssre.org
Cheryl Vince-Whitman, M.B.A., Ed.M., Senior Vice President, Education Development Center, Newton, MA USA cvincewhitman@edc.org
A Model for Using TechnicalA Model for Using TechnicalAssistance to Improve the Evaluation Assistance to Improve the Evaluation
Capacity of Local ProgramsCapacity of Local Programs
2
Agenda
• Brief Overview of Service to Science (STS).
• Evaluation Findings About Service to Science
• Key Characteristics of The Technical Assistance Provided By Service To Science
Overview of Overview of Service to Service to ScienceScienceInitiativeInitiative
4
The Need for Service-to-Science
• Federal agencies and others in the USA have promoted the use of evidence-based programs to prevent substance abuse.
• The supply of “proven” programs is too limited to meet many local conditions.
• There are many other programs that may work, but lack evidence of their effectiveness
5
Service-to-Science Goals
• To help innovative prevention interventions that address substance abuse (or related issues) develop, improve, and document evidence of their effectiveness.
• To increase the pool of effective and appropriate interventions.
6
Organization of Service-to-Science
• Funded by Center for Substance Abuse Prevention (CSAP) in 2004.
• Implemented through 5 regional prevention centers (Centers for the Application of Prevention Technologies – CAPTs).
• In 2007, CSAP funded a related initiative for programs designed for Native Americans. Implemented by the Native American Center for Excellence (NACE).
7
Service-to-Science Activities
• CAPTs solicit prevention program nominations from State prevention officials.
• CAPTs assess the readiness of candidate programs.
• CAPTs conduct regional Service to Science Academies.
• CAPTs provide follow-up technical assistance.
8
Service-to-Science Activities (Continued)
• Service to Science Academy participants – eligible to compete for “subcontract awards” to further enhance evaluation capacity.
• CAPTs provide limited technical assistance post subcontract award.
9
Number of Programs Served Through Service to Science
• As of July 2009, the CAPTs had provided 294 prevention programs with technical assistance.
• Since 2006, 110 programs have received the subcontracts of up to $30,000.
Selected Selected Evaluation Evaluation Findings Findings
11
Methods• Pilot study (2006) interviews and/or online survey
with program Directors. Target sample of 79 programs; 58 (73%) responded.
• Six case studies of 2005 STS participants and extended follow-up case studies on 5 that received mini-subcontracts. Six case studies of 2006 participants.
• 2007 online survey targeted 142 programs that had been participants for at least 15 months. 93 responded (a 77% rate after removing 21 programs for which respondents were no longer available).
Participant Participant ProgramProgram
Characteristics Characteristics
13
Primary Behaviors Targeted for Change
Percent (a)
Alcohol Use/Abuse 80%
Illicit Drug Use/Abuse 60%
Tobacco Use 58%
Violence (including bullying) 37%
HIV/AIDS 8%
(a) Multiple responses allowed. N=93.
14
Ages of Population(s) Served
Percent (a)
Adolescents (ages 13-17) 71%
Childhood (ages 6-12) 51%
Young Adults (ages 18-25) 19%
Early Childhood (ages 0-5) 10%
Adults (ages 26-55) 10%
Older Adults (ages >55) 7%
No Distinction by Age 3%
(a) Multiple responses allowed. N=93.
15
Race/Ethnicity of Population(s) Served
Percent (a)
No Distinction 85%
Hispanic or Latino 9%
American Indian or Native Alaskan 5%
Black or African American 4%
White 4%
Asian American 3% Native Hawaiian or Other Pacific Islander 3%
(a) Multiple responses allowed. N=93.
16
Strategies Used
Percent (a) Percent
Improve Knowledge / Awareness
90% Mobilize Community 27%
Improve Skills 85% Improve Identification and Referral
25%
Healthy Alternatives 79% Improve Policies 17%
Improve Norms 59% Improve Laws 12%
Improve Mental / Physical Health
54%Improve Access to and Quality of Care
11%
Build Collaboration 36% Improve Enforcement 8%
(a) Multiple responses allowed. N=93.
Service To Service To ScienceScience
Outcomes Outcomes
18
Value of STS ServicesPercent (a)
Satisfied with CAPT TA(Very Satisfied or Somewhat Satisfied vs. Somewhat Dissatisfied or Very Dissatisfied)
72%
CAPT TA Useful (A Little, Somewhat, or Very Useful vs. Not at All Useful)
91%
Would Recommend STS to Similar Programs (Definitely Yes or Probably Yes vs. Probably No or Definitely No)
91%
(a) N=93.
19
Factors That Facilitated Progress
When asked to discuss factors that facilitated progress on their evaluation the largest percentage of clients (79%) identified TA from the CAPTs.
(a) From the 2006 pilot study. N=58.
20
TA Contributed to Program Capacity(a) in 5 Most Common Issues
Percent (b)
Improving understanding of program rating systems such as NREPP & NASADAD
95%
Improving understanding of evaluation 95%
Developing and/or improving a logic model 92%
Developing and/or implementing a more rigorous evaluation design
95%
Improving organizational commitment to evaluation
84%
(a) Contributed “A Little,” “Somewhat,” or “A Great Deal” vs. “Did Not Contribute at All.” (b) N=93.
21
Seeking Recognition and/or Funding
Percent (a)
Plan to seek or are seeking recognition from NREPP or NASADAD
57%
Sought recognition from local, regional or other national organizations
13%
Prepared a journal article 13%
Presented evaluation findings at professional conference or meeting
13%
Used evaluation findings to help secure funding
23%
(a) N=93.
Defining Defining Features Of Features Of Service To Service To Science TAScience TA
23
Defining Features of Service to Science TA
• Client-centered
• Customized
• Long-term
• Relationship-based
• Flexible
Recommended