Upload
philomena-bates
View
236
Download
0
Tags:
Embed Size (px)
Citation preview
Kevin Baldwin, Ph.D.Sharon Johnson M.S.
Applied Research Services, Inc.Atlanta, GA
Kevin Baldwin, Ph.D.Sharon Johnson M.S.
Applied Research Services, Inc.Atlanta, GA
Sharing Your Court’s Successes: Practical
Evaluation of Accountability Courts
What is What is Evaluation?Evaluation?
• The systematic collection & analysis The systematic collection & analysis of information (data), often for the of information (data), often for the purpose of making decisions.purpose of making decisions.
• eeVALUEVALUEation – “value is our middle ation – “value is our middle name” – this implies that we are name” – this implies that we are assigning worth to somethingassigning worth to something
Why Evaluate?Why Evaluate?• To answer critical questions about To answer critical questions about
the courtthe court• To document the court’s processes To document the court’s processes
and demonstrate outcomesand demonstrate outcomes• To assess Fidelity of ImplementationTo assess Fidelity of Implementation• To comply with funder’s mandatesTo comply with funder’s mandates• To provide information and feedback To provide information and feedback
for continuous quality improvementfor continuous quality improvement
Presentation Presentation OutlineOutline• Process Evaluation Process Evaluation - the “who, what, where, - the “who, what, where,
when, how, and how much” associated with when, how, and how much” associated with delivery of a program or initiativedelivery of a program or initiative
• Fidelity of ImplementationFidelity of Implementation - the degree to - the degree to which a program or initiative is delivered as which a program or initiative is delivered as designeddesigned
• Outcome Evaluation Outcome Evaluation - the degree to which a - the degree to which a program or initiative achieves its stated program or initiative achieves its stated objectives – the “so what?” aspect objectives – the “so what?” aspect
• Cost Benefit Analysis Cost Benefit Analysis - the costs of your - the costs of your program and compares the costs (typically per program and compares the costs (typically per participant) to the cost of not having the programparticipant) to the cost of not having the program
Process Process EvaluationEvaluation
• Helps us understand why a program Helps us understand why a program was or was not successfulwas or was not successful
• Disappointing outcomes could be Disappointing outcomes could be attributed to an array of issues such as attributed to an array of issues such as poor program design, poor poor program design, poor implementation, failure to reach your implementation, failure to reach your target audience, etc.target audience, etc.
• Provides information for program Provides information for program replicationreplication
Logic ModelLogic Model• A logical model is a series of A logical model is a series of
statements that link the problems statements that link the problems your court is attempting to your court is attempting to address (conditions), how it will address (conditions), how it will address them (activities), and the address them (activities), and the expected results (immediate and expected results (immediate and intermediate outcomes, long-term intermediate outcomes, long-term goals). goals).
Copyright 2009, Applied Research Services, Inc.
Conditionsand Assumptions
Long Term GoalsIntermediate Outcomes Immediate OutcomesInterventions
Non-custodial parents(NCP) not meeting child
support obligations
NCPs going throughrepeated cycles of
non-payment,incarceration,
release from jail, andsubsequent non-payment
of child support
Current court treatmentfor child support
is often not effective
Children of delinquentNCPs suffer emotionally
and financially
CPs and NCPs sufferemotionally and
financially
Appropriate participantsare identified,
screened and enrolled
Relevant communityand agency stakeholders
unified as partners
NCPs undergocomprehensive
assessment of issues
Provide research-basedcomprehensive
substance abuse,mental health,
literacy, and employmentservices based onidentified needs
NCPs take parentingclasses, visitation, and monitoring to increaseparent-child bonding
Fully functioningChild Support Problem
Solving Courtin Carroll County
Increased collaborationamong agencies dealing
with NCPs
Increased ability toaccurately identify
NCP issues
Increased linkagesbetween NCPs andservice providers
Increased degree to whichparticipants attend
treatment, training andrelated services
Decreased time in jail dueto non-payment of
child support
Increased payment ofcurrent, past child support
CCCSC participantshave increased rates ofemployment, literacy,
increased earnings, andcompletion of GED
CCCSC participants havedecreased jail time and
associated court, jail costs
CCCSC participantshave increased rates and
frequency of childsupport payments
CCCSC participants havedecreased rates of positive
drug screens, self-reported substance abuse,increased levels of mental
health functioning
CCCSC participants haveincreased frequency
of positive interactionswith their children
Medical insurancecoverage for childrenof NCPs increases
CCCSC participantsreport increased
attachment to children
Carroll County hasan effective collaborative
means to addresschronic non-paymentof child support and
collection of arrearages
Children and parents have permanent and stable incomes and
living situations
Families are able tomeet the financial,
material, and emotionalneeds of their children
Children’s educationaland health needs are met
Parents report feelingthat their emotional
well-beinghas improved
Substance abuse andfunctional illiteracy
are drastically reduced
The intergenerationalcycle of addiction and
poverty is broken
AOC-CCCSC Logic Model – Revised 03/27/09
Provide intensivecase management
Judge-led court teamof key agency staff
Increased frequency,quality of parent-child
visitation
NCPs have a number ofunderlying issues that
contribute to non-paymentthat are not being
adequately addressed
Graduated system of incentives and sanctions
Legal services forlegitimation, mediation
Process EvaluationProcess Evaluation• Process evaluation provides a Process evaluation provides a
descriptive study of how your descriptive study of how your program was implemented and how program was implemented and how it operates nowit operates now
• It’s concerned with history, current It’s concerned with history, current operations, participant progress, operations, participant progress, obstacles and overcoming obstacles and overcoming impedimentsimpediments
Roehl & Guertin, 2000Roehl & Guertin, 2000
11 Key Questions A 11 Key Questions A Process Evaluation Should Process Evaluation Should
AnswerAnswer1.1. How was the program developed? (i.e. How was the program developed? (i.e.
aims, why initial policy/process aims, why initial policy/process decisions were made)decisions were made)
2.2. What are the policies & procedures of What are the policies & procedures of the court? How have they changed over the court? How have they changed over time & why? Include: selection criteria, time & why? Include: selection criteria, point of referral in the CJ system, point of referral in the CJ system, program requirements, sanctionsprogram requirements, sanctions
Roehl & Guertin, 2000Roehl & Guertin, 2000
11 Key Questions A 11 Key Questions A Process Evaluation Should Process Evaluation Should
AnswerAnswer3.3. Total eligible population for drug court? Total eligible population for drug court?
How are referrals & screenings How are referrals & screenings conducted? How many referrals are conducted? How many referrals are rejected & why?rejected & why?
4.4. Participant characteristics: demographics, Participant characteristics: demographics, criminal histories, SA/MH problems?criminal histories, SA/MH problems?
5.5. Available treatment & type of Available treatment & type of treatment/services received?treatment/services received?
Roehl & Guertin, 2000Roehl & Guertin, 2000
11 Key Questions A 11 Key Questions A Process Evaluation Should Process Evaluation Should
AnswerAnswer6.6. What happens to participants in drug court What happens to participants in drug court
(i.e. treatment, drug testing, sanctions)?(i.e. treatment, drug testing, sanctions)?
7.7. Who are the staff & what are their Who are the staff & what are their responsibilities? Annual budget & funding responsibilities? Annual budget & funding sources?sources?
8.8. What are the roles of the team – judge, What are the roles of the team – judge, prosecutor, attorneys, advisory board, etc.?prosecutor, attorneys, advisory board, etc.?
Roehl & Guertin, 2000Roehl & Guertin, 2000
11 Key Questions A 11 Key Questions A Process Evaluation Should Process Evaluation Should
AnswerAnswer9.9. What is the extent of collaboration What is the extent of collaboration
with other agencies such as with other agencies such as probation, parole, social services? probation, parole, social services? What information is shared between What information is shared between agencies?agencies?
10.10.What local court conditions affect What local court conditions affect your court (caseloads, community your court (caseloads, community attitudes)?attitudes)?
Roehl & Guertin, 2000Roehl & Guertin, 2000
11 Key Questions A 11 Key Questions A Process Evaluation Should Process Evaluation Should
AnswerAnswer
11.11.How long do participants stay in How long do participants stay in drug court? Who drops out, at drug court? Who drops out, at what point, and why? How many what point, and why? How many participants, with what participants, with what characteristics, graduate from characteristics, graduate from drug court?drug court?
Roehl & Guertin, 2000Roehl & Guertin, 2000
Sources of DataSources of Data• Meeting minutesMeeting minutes• Written mission statements, goals & Written mission statements, goals &
objectivesobjectives• Funding proposals, grant applicationsFunding proposals, grant applications• Annual reportsAnnual reports• Media (i.e. newspaper articles)Media (i.e. newspaper articles)• Caseload summariesCaseload summaries
Roehl & Guertin, 2000Roehl & Guertin, 2000
Sources of DataSources of Data• Program documentation (i.e. eligibility Program documentation (i.e. eligibility
criteria, program rules, roles/ criteria, program rules, roles/ responsibilities of key agencies, graduated responsibilities of key agencies, graduated sanctions, phase requirements, participant sanctions, phase requirements, participant contracts, promotional materials, budget contracts, promotional materials, budget documentsdocuments
• Interviews/focus groups with stakeholders, Interviews/focus groups with stakeholders, participants, etc.participants, etc.
• SurveysSurveys
Roehl & Guertin, 2000Roehl & Guertin, 2000
Sources of DataSources of Data• Automated data systemsAutomated data systems
– Demographic data (DOB, sex, race, ethnicity)Demographic data (DOB, sex, race, ethnicity)– Education (highest grade completed, degree earned)Education (highest grade completed, degree earned)– Employment status (FT, PT, student, unemployed)Employment status (FT, PT, student, unemployed)– Housing status (stable, unstable, homeless)Housing status (stable, unstable, homeless)– Community support (family, faith community, tx Community support (family, faith community, tx
community, work support)community, work support)– Mental health history (diagnosis, treatment history - # Mental health history (diagnosis, treatment history - #
inpatient & outpatient episodes, medication inpatient & outpatient episodes, medication compliance) compliance)
Sources of DataSources of Data– Substance abuse history (drug of choice, age Substance abuse history (drug of choice, age
began abuse, years of active substance abuse, began abuse, years of active substance abuse, # substance abuse treatment episodes) # substance abuse treatment episodes)
– Offense History (age at first arrest, nature & Offense History (age at first arrest, nature & type of charges, prior convictions, violent type of charges, prior convictions, violent convictions, previous jail episodes, previous convictions, previous jail episodes, previous prison episodes, previous probation/parole prison episodes, previous probation/parole episodes, current arrest/conviction – charges episodes, current arrest/conviction – charges & sentence)& sentence)
ResultsResults• Have all intended services been Have all intended services been
provided?provided?• Have the services been provided as Have the services been provided as
intended?intended?• What services not currently provided What services not currently provided
should be added to the program?should be added to the program?• Did the program reach the intended Did the program reach the intended
“target” population?“target” population?
Roehl & Guertin, 2000Roehl & Guertin, 2000
ResultsResults• Did the program widen the “net” of Did the program widen the “net” of
defendants who were supervised by defendants who were supervised by the court or who received CJ the court or who received CJ sanctions?sanctions?
• What problems were encountered in What problems were encountered in program implementation, operation program implementation, operation and performance?and performance?
• How were these problems resolved?How were these problems resolved?
Roehl & Guertin, 2000Roehl & Guertin, 2000
The Importance of The Importance of FidelityFidelity
• You could have the most powerful You could have the most powerful intervention ever devised, but it is intervention ever devised, but it is worthless if it is not delivered as worthless if it is not delivered as its developers intended. For its developers intended. For example, Excedrinexample, Excedrin®® works works wonders for headaches, but not wonders for headaches, but not when applied directly to the when applied directly to the forehead.forehead.
The Importance of The Importance of FidelityFidelity
• Your evaluation should explore Your evaluation should explore whether your program has been whether your program has been implemented as designedimplemented as designed
• Also, see how your implementation Also, see how your implementation compares to the Ten Key compares to the Ten Key Components for Drug Courts and Components for Drug Courts and our Georgia Standards for our Georgia Standards for Accountability CourtsAccountability Courts
https://www.ncjrs.gov/pdffiles1/bja/205621.pdf
Outcome EvaluationOutcome Evaluation• Measures the program’s influence on Measures the program’s influence on
factors such as graduation, recidivism, factors such as graduation, recidivism, abstinence, employment status, etc.abstinence, employment status, etc.
• Most useful when compared to a similar Most useful when compared to a similar group of persons that did not receive group of persons that did not receive programming (control or comparison programming (control or comparison group) – what outcomes would be group) – what outcomes would be expected without this program?expected without this program?
Evaluation Data, Evaluation Data, cont.cont.
• Evaluation has its own Evaluation has its own language, using words like language, using words like benchmarks, indicators, and benchmarks, indicators, and metrics to refer to datametrics to refer to data
• We also use phrases like “move We also use phrases like “move the needle”, “benchmarking”, the needle”, “benchmarking”, and “significance testing”and “significance testing”
Evaluation Data, Evaluation Data, cont.cont.
• The data to be collected derive The data to be collected derive directly from the questions we directly from the questions we are asking. It helps therefore to are asking. It helps therefore to craft questions of a directional craft questions of a directional nature:nature:• Do graduates have reduced (Do graduates have reduced () )
rates of recidivism?rates of recidivism?• Do participants have increased Do participants have increased
(() employment skills? ) employment skills?
Evaluation Data, Evaluation Data, cont.cont.
• We often cannot just report We often cannot just report however that something however that something increased or decreased. We increased or decreased. We need to say how much, and need to say how much, and compared to whom. Therefore compared to whom. Therefore we need benchmarks and/or we need benchmarks and/or control and/or comparison control and/or comparison groups.groups.
Evaluation Data, Evaluation Data, cont.cont.
• Comparing our results to a Comparing our results to a benchmark would be to say benchmark would be to say that our participants had a 5% that our participants had a 5% positive urinalysis rate, positive urinalysis rate, compared to the 25% rate compared to the 25% rate observed in non-participants.observed in non-participants.
Evaluation Data, Evaluation Data, cont.cont.
• A comparison group is a group of A comparison group is a group of people similar to our people similar to our intervention sample, but intervention sample, but receiving some other type of receiving some other type of intervention. A control group is a intervention. A control group is a group of people similar to our group of people similar to our intervention sample, but intervention sample, but receiving no intervention at all. receiving no intervention at all.
Outcome MeasuresOutcome Measures• Your logic model will provide you Your logic model will provide you
with some obvious areas for with some obvious areas for outcome measurementoutcome measurement
• Measure how well your program Measure how well your program met its goalsmet its goals
• Your outcome measures come Your outcome measures come from your objectives – they from your objectives – they indicate what you are trying to doindicate what you are trying to do
• They need to be measurableThey need to be measurable
Sample Data Sample Data SourcesSources• Official agency databases (i.e. GCIC, DFCS, Official agency databases (i.e. GCIC, DFCS,
hospital records) hospital records) • Drug test resultsDrug test results• Assessment resultsAssessment results• Treatment provider notes/reportsTreatment provider notes/reports• Pre/post testsPre/post tests• Interviews/focus groups with stakeholders, Interviews/focus groups with stakeholders,
participants, etc.participants, etc.• Surveys & other self-report dataSurveys & other self-report data
Sources of DataSources of Data• Automated data systems – combine process fields Automated data systems – combine process fields
with:with:– Program participation data: date referred, date entered Program participation data: date referred, date entered
program, # court hearings attended, types of treatment program, # court hearings attended, types of treatment completed (i.e. trauma group), # and nature of referrals completed (i.e. trauma group), # and nature of referrals made to other agencies/resources, Level of adaptive made to other agencies/resources, Level of adaptive functioning (i.e. can live independently, requires functioning (i.e. can live independently, requires supportive housing), degree of compliance with supportive housing), degree of compliance with medication, date left program, program completion status medication, date left program, program completion status (completed, did not complete, transferred, terminated) (completed, did not complete, transferred, terminated) and reason for any other status but completed.and reason for any other status but completed.
Sources of DataSources of Data
Outcome & performance measures: Outcome & performance measures: inpatient hospitalizations, crisis inpatient hospitalizations, crisis intervention episodes, ER visits, intervention episodes, ER visits, new arrests, new convictions, new new arrests, new convictions, new violations of probation/parole, new violations of probation/parole, new jail admissions, new prison jail admissions, new prison admissions, # failed drug testsadmissions, # failed drug tests
Comparison GroupComparison Group
• You will need data on your You will need data on your comparison group to compare comparison group to compare measures – take this into account measures – take this into account when determining the outcomes you when determining the outcomes you want to examine. How will you want to examine. How will you acquire the data you need from the acquire the data you need from the comparison group?comparison group?
Evaluation Data ExamplesEvaluation Data Examples• Outcome/performance data Outcome/performance data
examplesexamples• Number of graduatesNumber of graduates• Number of re-arrestsNumber of re-arrests• Percentage who relapsePercentage who relapse• Number of subsequent DFCS casesNumber of subsequent DFCS cases• Number of subsequent hospitalizationsNumber of subsequent hospitalizations• Number of subsequent ER admissionsNumber of subsequent ER admissions• Days cleanDays clean
Other Issues to ConsiderOther Issues to Consider• Timing & frequency of follow-up Timing & frequency of follow-up
• If your process includes post-participation If your process includes post-participation interviews, focus groups or surveys, be sure interviews, focus groups or surveys, be sure you have personal information to improve you have personal information to improve odds of keeping up with participants after odds of keeping up with participants after they have left the programthey have left the program
• Written consent for post-participation Written consent for post-participation follow-upfollow-up
• Respect the privacy & confidentiality of Respect the privacy & confidentiality of those with whom you are workingthose with whom you are working
• Federal laws (e.g., CFR 42, HIPPA) govern Federal laws (e.g., CFR 42, HIPPA) govern the use of substance abuse & health the use of substance abuse & health information – consider an IRB information – consider an IRB
• Respect and take into account the cultural, racial, ethnic and gender differences of your clients & their families
• Use results responsibly and ethically – don’t go beyond the intended use of the measures
• Federal laws (e.g., CFR 42, HIPPA) govern Federal laws (e.g., CFR 42, HIPPA) govern the use of substance abuse & health the use of substance abuse & health information – consider an IRB information – consider an IRB
• Respect and take into account the cultural, racial, ethnic and gender differences of your clients & their families
• Use results responsibly and ethically – don’t go beyond the intended use of the measures
Other Issues to ConsiderOther Issues to Consider
Cost-Benefit AnalysisCost-Benefit Analysis• Calculates the cost of a program vs. the Calculates the cost of a program vs. the
cost of the outcome to come up with a cost of the outcome to come up with a cost-benefit ratiocost-benefit ratio..
•Example: A study reveals that participants Example: A study reveals that participants of Drug Court A spend an average of 120 of Drug Court A spend an average of 120 less days in jail than similar offenders that less days in jail than similar offenders that do not go through drug court. do not go through drug court.
• Drug court participation to graduation Drug court participation to graduation = $3,800= $3,800
• 1 jail day = $45 x 120 days = $5,4001 jail day = $45 x 120 days = $5,400• Drug court savings = $1,600Drug court savings = $1,600
Cost Benefit Analysis Cost Benefit Analysis • Cost Benefit Analysis data Cost Benefit Analysis data
examplesexamples• Cost of treatment per participantCost of treatment per participant• Cost of day in jailCost of day in jail• Cost of foster care per childCost of foster care per child• Cost of adjudicating one felony Cost of adjudicating one felony
drug offensedrug offense• Cost of DFCS case investigationCost of DFCS case investigation
Evaluation Evaluation ResultsResults
• Ultimately your stakeholders and Ultimately your stakeholders and funders will ask these questions:funders will ask these questions:• Did it work? Was there an Did it work? Was there an
impact?impact?• How well did it work? How much How well did it work? How much
of an impact did you observe?of an impact did you observe?• How does this impact compare How does this impact compare
with results of alternative with results of alternative models? models?
Using Evaluation Using Evaluation FindingsFindings
• Describe your court and it’s Describe your court and it’s participantsparticipants
• Describe your court’s processes Describe your court’s processes and proceduresand procedures
• Continually improve your court’s Continually improve your court’s functioningfunctioning
• Document your impacts and Document your impacts and outcomes (sustainability) outcomes (sustainability)
Questions?Questions?
Presented at: 2013 AOC Accountability Court Conference – Atlanta, GA
Kevin Baldwin, Ph.D.Sharon Johnson, M.S.Applied Research Services, Inc.
404-881-1120 ext. [email protected]@ars-corp.com
www.ars-corp.com
Kevin Baldwin, Ph.D.Sharon Johnson, M.S.Applied Research Services, Inc.
404-881-1120 ext. [email protected]@ars-corp.com
www.ars-corp.com
Visit our web site at www.ars-corp.com or call (404) 881-1120