Governor’s Prevention Initiative for Youth: Evaluation

Jane A. Ungemack, Dr.P.H.

Evaluator

University of Connecticut Health Center

Evaluation

• Systematic efforts to collect and use information:– Document program implementation– Describe target populations/participants– Inform and improve program performance– Access program effectiveness– Increase accountability– Increase understanding

Governor’s Prevention Initiative for Youth

Long-term Goal: Reduce substance use among adolescents

Intermediate Goal: Reduce risk factors and increase protective factors for

substance use in the individual, peer, family, school and community domains

Target Group: 12-17 year old youth

Evaluation Framework OverviewII. Contextual Conditions

(Economic, Cultural, Risk Conditions, ATOD use)

2. State System 4. State-Level 6. State-Level 8. State-LevelCharacteristics/ Collaborative Intermediate Systems

Dynamics Strategies/ Outcomes ChangeActivities

Training/TA Training/TA

1. SIGMobilization

3. Sub-RecipientCharacteristics

Training/TA

5a. SubRecipient

Planning forScience-based

PreventionInterventions

7a. Sub-Recipient/CommunityImmediate

Outcomes & systemchanges

9. CommunityIntermediateOutcomes:Risk and

ProtectiveFactors

10. Long-termOutcomes:Behavioral

Impacts

5b. ProgramInterventions:

Activities/Actions

7b. ProgramOutcomes: Risk and

Protective Factor

Evaluation Approach

Process evaluation:

• documents program implementation and activities

Outcome evaluation:

• assesses program effects or impacts

Capacity-Building for Evaluation

Science-based approach

Evaluation and assessment as integral parts of the program design

– Community-level

– Program-level

Evaluation Team Relationship with Grantees

• Training and technical assistance

• Instruments and administration protocols

• Consultation and collaboration

• Statewide coordination

Grantee Responsibilities

• Develop a program plan based on the logic model • Specify measurable program objectives• Cooperate and collaborate with UConn Evaluation

Team• Coordinate community survey• Collect and submit process data • Collect and submit outcome data • Commit time and effort to evaluation activities

Assessing Community-Level Outcomes: CSAP Requirements

School survey

Use of core substance use, risk and protective factor measures

Community-Level Assessment: School Survey

• Mandated

• 7th-10th grade students

• Representative sample (minimum n=500; 125/grade level)

• Year 1 and Year 3

• First administration: February-April, 2000

School Survey

• Self-administered during a classroom period

• Anonymous and confidential

• Parental consents

• Sampling, instrument and administration protocols provided by UConn Evaluation Team

School Survey

Measures– Demographic characteristics– Lifetime and current use of ATOD– Risk and protective factors– Limited community-specific items

School Survey

Grantee responsibilities:– Planning/coordination with UConn Evaluation

Team– Planning/coordination with school personnel– Instrument duplication– Data cleaning– Data entry– Analysis

Assessing Program-Level Outcomes: CSAP Requirements

• Select a minimum of three programs (for each of three domains)

• Measure program outcomes using core measures

• Include a sufficient sample size for analysis

• Collect pre- and post-test data

Program-Level Evaluation

Process evaluation:• Document program implementation and activities

Outcome evaluation: • Assess program outcomes

Program-Level Evaluation: Process Evaluation

Each program will be responsible for reporting:– Prevention strategies– Types of activities– Dosage– Number served– Participant characteristics (age, gender,

race/ethnicity, etc.).

Process Evaluation

• Minimum Data Set (MDS)

• Instruments, protocols, and training provided by the UConn Evaluation Team

Program-Level Evaluation:Outcome Evaluation

• Based on the logic model, identify measurable objectives that you will address in your program

• Program objectives should be selected from one or more of the risk/protective factors included in the RFP list of Connecticut Intermediate Outcomes

Outcome Evaluation

• UConn Evaluation Team staff will work with each grantee to finalize program-specific objectives and measures

• All grantees will be asked to participate in a pre/post-test assessment of age-eligible participants as appropriate

Outcome Evaluation

Pre- and Post-Test Assessments

• Youth participation will be voluntary

• Confidential

• Informed consents

• Standardized instrument plus optional program-specific items

• Minimum sample size = 50

Considerations for Estimating Evaluation Costs

• Personnel (.25 FTE minimum recommended)

• Computer equipment

• Photocopying

• Office supplies

• Data collection and cleaning

• Data entry

Evaluation Themes for the Governor’s Prevention Initiative

• Evaluation at all levels

• Science-based

• Capacity-building

• Collaboration

• Coordination

• Sustainability