View
216
Download
2
Category
Tags:
Preview:
Citation preview
SOUTH DAKOTA’S SFP SIG EVALUATION
Population Health Institute - University of Wisconsin - Madison
1
Doug Piper, Senior ScientistPopulation Health Institute
School of Medicine and Public HealthUniversity of Wisconsin – Madison
SPF SIG Advisory Council MeetingChamberlain, South Dakota
September 20, 2010
Intention of the SPF SIG Project
Population Health Institute - University of Wisconsin - Madison
2
The SPF SIG Project was designed by the folks at CSAP to change the way substance abuse prevention is done in the states.Emphasizing data driven planning Evidence-based programs Evaluation•Complex•Resource intensive•Largely driven by CSAP requirements
Population Health Institute - University of Wisconsin - Madison
3
Expectations for State and Local
SPF SIG Grantees Fidelity implementing the SPF process at the state, tribal and coalitions level will increase prevention capacity and a build a stronger prevention infrastructure at all levels.
“Infrastructure” is the underlying foundation or basic framework which is necessary for an organization or system to function.
The components of strong prevention infrastructure might include: A unifying theory of prevention;
Effective leadership, Readiness to collaborate organizations and
agencies;
Population Health Institute - University of Wisconsin - Madison
4
Expectations for State and Local
SPF SIG Grantees:Components of a strong
Infrastructure (cont.)
Population Health Institute - University of Wisconsin - Madison
5
Increased prevention capacity & stronger prevention infrastructure at the local level will lead to…. • Community-level prevention system changes independent of the direct allocation of SPF SIG funds.• Delivery of evidence-based, culturally appropriate direct service and environmental prevention programs.• Further refinement and efficiency of planning and implementation.
Expectations for State and Local
SPF SIG Grantees
Population Health Institute - University of Wisconsin - Madison
6
The delivery of evidence-based programs implemented with fidelity will lead to improvements in community level and individual outcomes. Integration of cultural competency principals and practices at the state and local levels will promote sustained prevention capacity and infrastructure development.
Expectations for State and Local
SPF SIG Grantees
Population Health Institute - University of Wisconsin - Madison 7
South Dakota’s AOD Prevention
System
Funding sourceSD governmental unitProgramsState-wide advisory bodyMOUs with DADA to run programs
Population Health Institute - University of Wisconsin - Madison
8
South Dakota SPF SIG Goals for the Local
CoalitionsThe South Dakota SPF SIG Advisory Committee, based on extensive epidemiological data, have chosen the following goals:
1.Reduce the incidence and consequences of underage drinking in South Dakota among youth ages 12 to 20 years.
2.Reduce the incidence and consequences of binge drinking among 18 to 25 year olds in South Dakota.
Population Health Institute - University of Wisconsin - Madison
9
South Dakota’s 30 day use for alcohol and binge drinking for 18 to 25 year olds
compared to National data
NSDUH data
Population Health Institute - University of Wisconsin - Madison
10
Primary Evaluation Objectives
1.Describe the process through which the SPF project is implemented in South Dakota and at the community and tribal levels.2.Measure prevention capacity and infrastructure development in South Dakota and the community and tribal levels.3.Determine to what extent the SPF has produced changes in targeted communities and individual outcomes.
Population Health Institute - University of Wisconsin - Madison
11
Evaluation Types1.Monitoring is checking records and doing site visits to ensure that the intervention included in the contractual work plan is being implemented with reasonable fidelity. This is DADA’s role and not that of the evaluation team.2.Process evaluation documents the program implemented by tracking the input or activities. It defines the independent variable.
3. Outcome evaluation measures the program’s effects that you expect to achieve after the program is implemented
Program actually
implementedInputs Outcomes –
targeted behavior
Population Health Institute - University of Wisconsin - Madison
12
• Attempt to change the surroundings in which people choose to use, or not use, substances. • Aim at changing environmental conditions that contribute to substance abuse. Accessibility of substances, Law enforcement efforts to reduce illegal activity, penalties for illegal activity, as well as Social norms regarding substance use.
• Often involve changing laws, policies, and regulations (LPRs) to make it more difficult for people to abuse substances. • Intended to change individual-level behavior by creating environments that dissuade substance abuse.
Environmental Strategies
13
Logic Models and Evaluation* ENVIRONMENTAL EXAMPLE 1
South Dakota SPF SIG Priorities: (check one)*__Underage drinking among 12 to 20 year olds. __Binge drinking
among 18 to 25 year olds.
Logical connection
Substance-related
consequences
Intervening variables & underlying conditions
StrategiesOutcome evaluation
indicators
Car crashes among young adults
Few officers assigned to enforce alcohol & DUI laws
Work with local law enforcement to increase surveillance at bar time
Law enforcement records show an increase in patrols at bar time
Community norms accept heavy drinking
Media advocacy to increase community concern about drinking and driving
Community survey of 18 to 25 yr olds show a reduction in drinking and driving
Ban alcohol price promotions & happy hours
Passage of new ordinances or agreements to ban or reduce alcohol promotions
* If you are targeting both priorities you must complete a separate Logic Model for each one.
14
Logic Models and Evaluation* NVIRONMENTAL EXAMPLE 2
South Dakota SPF SIG Priorities: (check one)__Underage drinking among 12 to 20 year olds. __Binge drinking
among 18 to 25 year olds.
Logical connection
Substance-related
consequences
Intervening variables & underlying conditions
StrategiesOutcome
evaluation indicators
Disruptive beer parities among high school students
Parents don’t monitor parties or alcohol supplies
Social event monitoring and enforcement
Existing student survey shows overall reduction in use among 18 to 25
Students arrive at school after having drunk at lunch time
Easy social availability of alcohol (adults provide alcohol)
Stiffen school policies regarding alcohol use among students
Number of alcohol-related school suspensions decreases
Retailers sell to under-aged youth
Enforce under-age retail sales laws
Proportion of negative compliance checks decreases
* If you are targeting both priorities you must complete a separate Logic Model for each one.
Individual-focused Prevention
Population Health Institute - University of Wisconsin - Madison
15
This approach to prevention endeavors to change individuals within the community or society. Their surrounding environment is not a target for change to any great degree. They may aim to improve an individual’s refusal skills, increase their connection to the community or promote positive family interaction. Examples are: School based programs Indicated programs which focus on high-risk individuals
16
Logic Models and Evaluation INDIVIDUAL-BASED PROGRAM EXAMPLE
South Dakota SPF SIG Priorities: (check one)*
__Underage drinking among 12 to 20 year olds. __Binge drinking among 18 to 25 year olds.
Logical connection
Substance-related
consequences
Intervening variables & underlying conditions
StrategiesOutcome evaluation
indicators
Surveys or other epidemiological data show unacceptable rates of underage use
Drinking is considered a rite of passage. “I did it when I was in high school.”
Implement a school-based evidence-based program.
All participants in individually-focused programs must be surveyed with NOMs questions.
Easy social availability of alcohol (adults provide alcohol)
School policies regarding alcohol use among students
Number of alcohol-related school suspensions declines
* If you are targeting both priorities you must complete a separate Logic Model for each one.
Population Health Institute - University of Wisconsin - Madison
17
Challenges for Evaluating Community Outcomes of Environmental
Strategies Dearth of data corresponding to the Coalition’s service area and the project’s target populations.Sensitivity of the data to our interventions, e.g. alcohol-related car crash deaths. Immediacy of the intervention’s affects on the target population. Dosage of the intervention penetrating to the target population. May require additional data collection for 20 to 25 year olds for social norms, etc.
Challenges for Outcome Evaluation of Individual-based
Prevention
Population Health Institute - University of Wisconsin - Madison
18
Need a comparison group to show more than just change. CSAP requires all participants in individual-focused programs to be surveyed:• Must include NOMs questions• Data collection methodology must be up to standards• Periodicity of the surveys must be approximately the same. Local Coalitions may be able to secure permission from local school districts to use their data. Only relevant to the underaged drinking target population of 12 to 18 year olds.
Evaluation Tasks for the Coalitions (1 of 3)
Population Health Institute - University of Wisconsin - Madison
19
PROCESS EVALUATION
Phase
1
Phase
2Timeline
Data collected by CoalitionsCommunity-Level Instrument (pt 1) – Community Partner Activities – Coalitions report on CSAPs MRT web-based reporting system
X XAnnually – November 1st
Community-Level Instrument (pt 2) – Intervention Strategies – Coalitions report on CSAPs MRT web-based reporting system
XTwice a year – Nov. 1st & May 1st
Coalitions report progress on SD’s Prevention Management System (KIT data-base)
X Monthly
Coalitions provide Data Collection Reports of what data was collected, how it was collected and by whom.
XAnnually starting June 1, 2012.
All materials developed by the Coalitions for the SPF SIG project will be forwarded to UW PHI for archives
X XAs appropriate
Coalitions provide data as respondents
Coalition Capacity Survey – Coalitions interviewed by UW Population Health Institute
X X
December 2010 & September 2015
Evaluation Tasks for the Coalitions (2 of 3)
Population Health Institute - University of Wisconsin - Madison
20
OUTCOME EVALUATION
Phase
1
Phase
2Timeline
Data collected by coalitions on individually-focused interventions
Surveys of all participants in individually-focus interventions with the Participant-Level Instrument (PLI) to be developed by UW PHI (including NOMs).
X
Pretest and post-test of all participants
Data collected by coalitions on environmentally-focused interventions
Existing epidemiological data appropriate to intervention,# killed in alcohol-related auto crashes% of auto crashes that are alcohol-relatedUnderage drinking arrestsDUI arrests of persons 21 to 25 yrs old.
X
Submit when data is available, at least annually
Evaluation Tasks for the Coalitions (3 of 3)
Population Health Institute - University of Wisconsin - Madison
21
EVALUATION TRAINING AND TA
Phase
1
Phase
2Timeline
Respond to Needs Assessment Survey of evaluation skills of Coalitions conducted by UW PHI
XOctober 2010
Participate in three formal evaluation trainings
XOct. 4 – 6, 2010 plus 2 dates TBD
Participate in quarterly Learning Communities
XStarting in July 2011
Request TA on evaluation issues from UW PHI
X X As needed
Participate in evaluation site visits X Twice a year
Suggestions for recruiting an on-site evaluator for the South
Dakota SPF SIG Project to be a member of the evaluation team,
housed in Pierre and an employee of the University of Wisconsin –
Population Health Institute.
Population Health Institute - University of Wisconsin - Madison
22
Population Health Institute - University of Wisconsin - Madison
23
Doug Piper5901 Research Park Blvd.Madison, WI 53719
608.263.7629DPiper2@wisc.edu
Recommended