Embed Size (px)
Citation preview
1
Evidence Based Practice:One Agency’s Journey
Dr. Shirley Hunt
Director, Research Division
Orange County Probation Department
Presented to
American Probation and Parole Association
February 2, 2010
Orange
County
CA
2
OC FAST FACTS
• 3rd largest county in CA (~3.1 million)
• Exceptional wealth and extreme poverty
within 798 square miles
• County oversight of probation (no direct
ties to state)
• Supervision of adults (~ 16,000) and
juveniles (~6,000)
3
OC Probation’s EBP Journey
• 1980s Decade – Implemented NIC Probation Model
• Validated Risk-Need Assessment Instruments • Initiated Supervision by risk & Case Plan Objectives by needs• Began tracking workload & staffing (paper and pencil)• Set up a PC-database for assessments
• 1990s Decade – Building & Expanding Capacity• Growth in use of OC data to inform strategic planning and fiscal decisions• Began formal program evaluations (“8%” repeat offenders)• Implemented automated “Case Management System” (Individual-level +
Unit workload)
• 2000s Decade – Revisiting Goals & EBP • Expanded internal research capacity • Continued use of OC data + “what works” to inform OCPD Operations• 2008 – selected as 4th NIC EBP site• Expansion of automated case management system (“windows-based”)
4
Measuring EBP ……
from the Inside/Out
• What data do you have now? What do
you need?
• Where is it? How will you get it?
• Once you have it, what do you do with it?
5
What Data?
“Assess agency data using EBP template”
• Demographic • Age (DOB), gender, race/ethnicity, residence, citizenship
• Supervision • Presenting offense, conviction date, supervision start date • Risk/Need assessment scores (changes over time – overall/subgroups)• Changes in supervision type and/or level (dates occurred)• Officer-Offender Interaction (contacts - #, type, quality of..)
• Treatment/Sanctions• Treatment Type/Dosage (agency, referral date, completion status, exit date)• Sanctions placed (causal event, date, custody days imposed, other,
completion status)
• Outcomes • Arrests, violations, incarceration (type of offense, dates)• Length of supervision (probation termination date, type)
6
Where are the Data?
“Map your system to identify key points of service”
• What information is available at each point?• Who is the “holder” of the data? One agency/multiple agencies.
• Probation/Court/DA/Social Services/Health Care/Treatment Provider/• What additional data can/should be captured at each point?
• How will you access the data? • How is the existing data currently maintained (paper; automated, etc.)• What barriers exist to exchanging or sharing of data? (confidentiality,
technology, etc.)
• How will you track information over time?• What’s needed for a comprehensive “data record” from entrance to exit• What data elements must be permanent (never overridden)
• What is the plan to ensure reliability and validity of information over time? (very important question!)• Who will be the “responsible” owner?
7
Challenges: Creating an EBP data record
Low-Risk High-Risk� 49 yr. old male convicted driving
under the influence (heroin)
� 2 prior convictions (DUI-alcohol)
� Assessed “High risk/Low Needs”
� Risk: Freq. moves; Unstable
emplymt; Serious drug/alcohol
us; Priors; < 19 at lst offense;
� Needs: Lower skill level; unsatisf.
Emplmt; Situational finan. issues
� Supervision: Prop 36 -> Reg Sup
-> Drug court
� Treatment: 4 alco/drug
treatment programs (failed)
� TERMINATED PRISON:
THREE TECH. VIOLATIONS
� 43 yr. old male convicted driving
under the influence of meth.
� No prior convictions
� Initially placed on Informal/cont’d
meth use -> Formal/Prop 36
� Assessed “Low risk/Low needs”
� Risk: (1) recent unstable
employment - emplyd 11 years until
incarcerated; (2) serious drug abuse
� Needs: (1) Drug problem; (2)
Emot. issues (abused by alcoholic
father; ADHD); (3) Finan. issues
� Min. Sup./ Enrolled in drug trtmt
� TERMINATED NO NLV;
COMPLETED TREATMENT
8
How do you use data to measure
progress and Improve Outcomes?
(or)Now you have the data, what do you
do with it?
“ Begin asking/answering this
question from day one and
continuously revisit”
9
EBP MEASURES:
INFORMATION/ANALYSIS MATRIX
COMMUNITY OFFENDER OFFENDER
PROTECTION ACCOUNT. COMPETENCY
PROCESS
INTERMEDIATE
LONG-TERM
Short-term measures of progress
Long-term measures of success in
meeting goals
Counting of numbers
10
……..PROCESS COUNTING…….
COMMUNITY OFFENDER OFFENDER
PROTECTION ACCOUNT. COMPETENCY
• # on probation
by risk level
• # of contacts by
risk level
• # of prob. Out
to warrants
• # of prob. drug
tested
administered
• # of prob. With
positive drug
results
• # referred to
drug counseling
by risk level
• # of juv prob
attending school
regularly
What are we doing? OCPD
____________
11
Unit Summary Report (Example)
GSUSUM1UNIT STATISTICAL
SUMMARYPage: 1
5/11/04 9:14:27
Unit: 059 JUVENILE DECENTRALIZED IN
Month: April 2004
Supervisor:
(Cases) Total
Total Reso Prog W/L
Off # Officer Name Cases New High Med Low Unc Cus FM Ref PT Dispo Rpt Points
0591 58 30 28 112.48
0592 57 35 22 116.72
**Report Totals** 115 65 50 229.2
CASE MOVEMENTSTATISTICS
1. In Cases: New- 12 Transfer- 0
2. Out Cases: Trm- 11 Transfer- 3
Warrants- 0
3. % Cases Turn Over- 22.61
12
Intermediate Measures
• % of population with completed r/n assessment (set goal)
• Gain scores: Changes in protective measures
• # & type of technical violations & dispositions
• % of medium/high risk offenders with case plans developed within specified timeframe
• % of case plans that address the top 3 identified criminogenicneeds based on assessment
• Average length of supervision sorted by risk level.
13
Employment Status:
Percent with satisfactory employment
Intermediate measure example
28%
40%
0%
10%
20%
30%
40%
50%
1% at Entry % at One Year
14
Technical Violations at First Reassessment:
% With No Tech. Violations by Initial Risk Level
Intermediate measure example – Opportunity to do QAS
70%
83%92%
0%
20%
40%
60%
80%
100%
High Medium Low
Initial Risk Classification
% W
ith
No
Tech
Vio
lati
on
s
15
Long-Term Outcomes
How well are we doing?
1. % Terminating Satisfactorily
2. # of probationers terminating with no new law
violations (while under probation supervision?
3. Violation outcomes by initial risk level/scores
16
Adult Termination Outcomes FY 08/09
34%30%
12%17%
7%
0%
10%
20%
30%
40%
50%
Normal Prison/Jail Cond/ROS Early Term Other Term
FY 08/09 Adult Termination Status
17
Terminations with Zero New Law:
Ten Year Outcomes
Adult Probationers Terminating
w ith No New Law Violations
69%71%
69%71%
66%
71%69%
67%66%
74%
56%
60%
64%
68%
72%
76%
80%
FY 99-
00FY 0
0-01
FY 01-
02FY 0
2-03
FY 03-
04FY 0
4-05
FY 05-
06FY 0
6-07
FY 07-
08FY 0
8-09
% o
f T
erm
inate
d P
rob
ati
on
ers
18
Violation Outcomes by Initial Risk
38%
23%
39%
67%
12%
20%
87%
5%8%
0%
20%
40%
60%
80%
100%
High Medium Low
Initial Risk Classification
Adult FY 08/09 Terminations:
Violation Outcomes by Initial Risk level
New Law
Tech Only
None
19
Adult Terminations with New Law Violation by Initial Risk Score
(2008-09 Terminations, n=4292)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
Total Initial Risk Score
% 1
+ N
LV
Medium HighLow
20
OC Probation’s EBP Journey:
What have we learned?
• What have we done well?• Maintaining formal risk/need assessment as cornerstone of operations
for 2+ decades (supervision & case plan)
• Integrating research/evidence (data) into organizational fabric
• Automating functions and production of process reports
• Focusing on agency-level performance measures
• What are we doing….medium?• Case plans are “barebones”
• Minimal attention to quality control (e.g. interrater reliability)
• Measures of officer performance still traditional (i.e. contacts, testing)
• Limited use of data to show outcomes at the officer/unit level
• What still remains to be done……?
21
2010’s Decade:
Coming EBP attractions in OCPD
• Building next generation of continuous quality improvement
• Developing automated outcome reports
• Expanding performance measures (from agency-wide out to staff level)
• Strengthening Measurement of “officer-offender”relationship quality
22
For more information
• Dr. Shirley Hunt
Orange County Probation DepartmentPO Box 10260Santa Ana, CA [email protected]
OrMicheal CollinsOrange County Probation [email protected]
Orange
County
CA
