SB81/ROCKWhat Makes Evaluation So Important?

September 18, 2008Prepared by Urban Strategies CouncilEvaluation TeamSteve Spiker, Director of ResearchBill Heiser, Research and Program AssociateJunious Williams, CEO

www.urbanstrategies.org

California Probation Parole and Correctional Association

77th Annual Training Conference

Outline

Presenter Biographies The Council What is Program Evaluation Why Evaluate? The ROCK Program Evaluation Outcomes

Urban Strategies Council Non-profit founded in 1987 in Oakland,

California Mission is the elimination of persistent poverty

by building vibrant, healthy communities Operating Programs:

Economic Opportunity Education Excellence Community Safety & Justice

Support Programs: Research & Technology Community Capacity Building

Community Safety & Justice

Current Projects Alameda County Reentry Network

(www.acreentry.org) Reentry Health Gap Survey Oakland Community Policing Oakland Crime & Homicide Analysis Alameda County Violence Prevention Initiative Community Service Gateway Reentry ProjectPast Projects Reentry Health Task Force Richmond Violent Crime Analysis

What is Program Evaluation Formalized approach to studying processes

and impacts Include both quantitative and qualitative

methods In more recent years evaluation focuses on

utility, relevance and practicality and less on scientific validity

Focus points: What do you need to know? How can it be applied?

What is Program Evaluation 5 dimensions:

1. Needs Assessment (Why do an evaluation, what must we know?)

2. Program Theory (How does the program work?)3. Process Analysis (Formative Evaluation)4. Impact Analysis (Goals met?)5. Cost-Benefit & Cost-Effectiveness analysis (Is

what we did scalable beyond a pilot?)

Why Evaluate?1. Program improvement2. Comparison between programs3. Demonstrate measureable impact of new

programs4. Decision making- are there particular groups

who will most benefit?5. Independent verification of program

suitability and success

Evaluation Considerations1. Why are we thinking of an evaluation?2. Who is the evaluation for? (Audience)3. What type of information will we need?

(Data)4. Goals, processes or outcomes?

5. Evaluators can help improve program design6. Evaluators can assist in building skills,

knowledge and abilities of staff7. Include them as part of your project team,

not as outsiders

Evaluation Issues1. Are you asking the right questions?2. There may be multiple “right” answers3. Is there a broader context you need to

consider?4. Need to include stumbling blocks and

failures!5. Consider assumptions in model/theory6. How well can we generalize our results?

When to hire an evaluator? Ideally at the proposal-writing stage! Preferably before you start providing services Never wait till the first annual reports are due!

SB81/ROCK Evaluation Formative Evaluation Data Collection & Instrument Design Data Analysis Causation & Recommendations

Population Characteristics Total probationers 18-24 in Oakland/Hayward:

264 228 Male (84%), 36 Female (14%)

SB81/ROCK Evaluation Performance goals to be measured:

1. Enrollment in recommended programs and services;

2. Participation in the required or recommended services;

3. Acquisition of the intended knowledge or skills; 4. Completion of the program; and 5. Achieving the intended purpose of the program

such as GED, employment, cognitive behavioral change, effective anger management, reduced and more effectively managed conflict, etc.

Formative Evaluation 6 Months in length Refining program definitions, hypotheses and

procedures Finalize specific outcome and results

measures Clarify and validate selection procedures Baseline data analysis

Process Documentation In order to evaluate a new program we must

know (i.e. document) the following: Procedures and processes: what happens in

certain situations Referrals: how, when and why they are made Theory of change- why each part will have an

impact Contextual and environmental conditions affecting

the program implementation and outcomes

Data Collection1. Full probation population demographics2. LS/CMI Assessment scores3. Referrals: when, by whom, why and how well

they worked4. New arrests and convictions5. Probation violations, revocations6. Program removal- moving out of county, death7. Surveys from pre/post cognitive behavioral

classes8. Interviews with DPOs on program

implementation and processes

Evaluation Outcomes Causation: are the outcomes for the target

group statistically different from those in the comparison group?

Can we definitively say that this model of services reduces recidivism to state prison?

If yes, then how well does this model suit the general probation population in Alameda County (or the State?)

Follow Up For more information on:

The ROCK evaluation.. Other evaluation services.. Mapping services or spatial analysis.. Analytical services.. Program design..

Please contact us: Steve Spiker, Research Director

steves@urbanstrategies.org or 510-893-2404