Evaluation IIEvaluation IIOutcomesOutcomes

Cost benefit & Cost benefit & effectivenesseffectiveness

OutcomesOutcomes

Dependent on the goalsDependent on the goals Multiple goals, do they competeMultiple goals, do they compete Multiple stakeholdersMultiple stakeholders Development of measurable Development of measurable

outcomes that reflect program goalsoutcomes that reflect program goals If the measurements do not reflect If the measurements do not reflect

the goals, then they are not usefulthe goals, then they are not useful

OutcomesOutcomes

A goal may have one than one A goal may have one than one measure (indicator)measure (indicator)

In evaluation research there are In evaluation research there are often multiple measures of several often multiple measures of several outcomesoutcomes

Need to identify the IV and then Need to identify the IV and then multiple measures of the DVmultiple measures of the DV

Example: Manhattan Bail ProjectExample: Manhattan Bail Project

Use of a prediction instrument to Use of a prediction instrument to release arrestees on their own release arrestees on their own recognizancerecognizance

Goals: relieve jail overcrowding, Goals: relieve jail overcrowding, presence at trial, protect public presence at trial, protect public safety, decrease costssafety, decrease costs

Measures: % RORMeasures: % ROR % that appear at hearing % that appear at hearing

ExampleExample

% released ROR that abscond or % released ROR that abscond or commit new crimes (public safety)commit new crimes (public safety)

Costs of the program minus the Costs of the program minus the amount saved by successful RORsamount saved by successful RORs

Might be successful in one sense but Might be successful in one sense but not in anothernot in another

Evaluation designsEvaluation designs

Experiment: randomly assign Experiment: randomly assign subjects to different treatment subjects to different treatment groups, i.e., intervention/no groups, i.e., intervention/no interventionintervention

Example: random assignment to EM Example: random assignment to EM with home detention and home with home detention and home detention without EM (control)detention without EM (control)

Logic of experimentsLogic of experiments

Experiments Experiments

Must ensure that the treatment is Must ensure that the treatment is delivered, i.e., is home detention delivered, i.e., is home detention enforced? Is the EM equipment enforced? Is the EM equipment working properly? This is a matter of working properly? This is a matter of monitoring, and must be established monitoring, and must be established before evaluating treatment effectsbefore evaluating treatment effects

What is the DV? How will outcomes What is the DV? How will outcomes be measured? Rearrest rates? be measured? Rearrest rates? Other? Depends on goalsOther? Depends on goals

ExperimentsExperiments

Treatment integrity and program Treatment integrity and program delivery are important—if they are delivery are important—if they are problematic, then outcomes cannot problematic, then outcomes cannot be evaluatedbe evaluated

Monitoring, making sure that Monitoring, making sure that program is being followedprogram is being followed

Quasi-experimentsQuasi-experiments

Random assignment is often not Random assignment is often not possiblepossible

Sometimes feasible for Sometimes feasible for programs/services, but may be programs/services, but may be restricted for legal or political restricted for legal or political reasonsreasons

Random assignment typically not Random assignment typically not possible for evaluation of laws or possible for evaluation of laws or some policiessome policies

Quasi-experimentsQuasi-experiments

Due process, equal protection issuesDue process, equal protection issues Example: drug court participants Example: drug court participants

could probably not be randomly could probably not be randomly assigned, must look at the group as a assigned, must look at the group as a wholewhole

Violates the logic of experiments—is Violates the logic of experiments—is any change in the DV due to the IV, any change in the DV due to the IV, or to something else (extraneous)? or to something else (extraneous)?

Quasi-experimentsQuasi-experiments

Ex post evaluations: an evaluation Ex post evaluations: an evaluation sometime after a program has gone sometime after a program has gone into effectinto effect

Full coverage programs—everyone is Full coverage programs—everyone is subjected to some treatment/IV, subjected to some treatment/IV, there is no controlthere is no control

Example: graduated license law in Example: graduated license law in Missouri (how to evaluate)Missouri (how to evaluate)

Quasi-experimentsQuasi-experiments

Treatment units such as Treatment units such as neighborhoodsneighborhoods

IV in some areas, not in areasIV in some areas, not in areas Uncontrolled variables, such as the Uncontrolled variables, such as the

actions of individuals within the actions of individuals within the neighborhoodsneighborhoods

Target and comparison areasTarget and comparison areas Usually fear of crime and crime ratesUsually fear of crime and crime rates

Quasi-experimentsQuasi-experiments

Non-equivalent control group designsNon-equivalent control group designs No random assignmentNo random assignment Attempt to select an experimental Attempt to select an experimental

and comparison group that are as and comparison group that are as similar as possiblesimilar as possible

Can assess this by measuring both Can assess this by measuring both groups on important variables and groups on important variables and determining if there are differencesdetermining if there are differences

Quasi-experimentsQuasi-experiments

Time series designsTime series designs Interrupted time series designsInterrupted time series designs Measure before and after an Measure before and after an

intervention is introducedintervention is introduced Repeat in other settings, other times, Repeat in other settings, other times,

other subjectsother subjects Single series design—introduce and Single series design—introduce and

then remove the interventionthen remove the intervention

Quasi-experimentsQuasi-experiments

Time series design with switching Time series design with switching replicationsreplications

Combination of the two—can do a Combination of the two—can do a single series design in one location single series design in one location and then introduce it in anotherand then introduce it in another

Example: helmet laws lead to Example: helmet laws lead to reduction in motorcycle theftreduction in motorcycle theft

Checked for displacement to cars, Checked for displacement to cars, bikesbikes

Cost benefitCost benefit

Costs of the intervention compared Costs of the intervention compared to the money potentially savedto the money potentially saved

Manhattan bail project, costs of the Manhattan bail project, costs of the program compared to money saved program compared to money saved when the arrestees are not jailed when the arrestees are not jailed (costs of jail for estimated specified (costs of jail for estimated specified periods of time)periods of time)

Other potential savings (wages, Other potential savings (wages, taxes)taxes)

Cost benefitCost benefit

Sometimes the cost benefits may be Sometimes the cost benefits may be calculated on a long term basiscalculated on a long term basis

Unfortunately the government is not Unfortunately the government is not necessarily responsive to this necessarily responsive to this argument, as they operate on a more argument, as they operate on a more short term basisshort term basis

Example: costs of corrections Example: costs of corrections (prisons) vs community corrections (prisons) vs community corrections

Cost benefitCost benefit

Boot camp evaluations: aside from Boot camp evaluations: aside from other outcomes, (i.e., recidivism), do other outcomes, (i.e., recidivism), do boot camps save money?)boot camps save money?)

Are inmates Are inmates deterred/rehabilitated/punished in a deterred/rehabilitated/punished in a shorter period of time?shorter period of time?

More inmates could be processed at More inmates could be processed at a faster, less expensive ratea faster, less expensive rate

Cost benefitCost benefit

In this example, note the potential In this example, note the potential dangers of net wideningdangers of net widening

Is money saved by performing the Is money saved by performing the intervention in the long run?intervention in the long run?

Major argument that advocates of Major argument that advocates of prevention programs must use in prevention programs must use in order to sell programsorder to sell programs

Cost effectivenessCost effectiveness

Costs of the program compared to Costs of the program compared to the non-monetary outcomesthe non-monetary outcomes

A program might be cheaper but A program might be cheaper but outcomes might not be better (or outcomes might not be better (or even worse, or with unanticipated even worse, or with unanticipated unwanted side effects)unwanted side effects)

Community corrections: cheaper, Community corrections: cheaper, recidivism rates not different recidivism rates not different

Cost effectivenessCost effectiveness

If recidivism is not higher, is the If recidivism is not higher, is the program effective? What are the program effective? What are the other issues?other issues?

Punitiveness or deterrence Punitiveness or deterrence (individual, general)? Effect on (individual, general)? Effect on inmates who avoid prisonization? inmates who avoid prisonization? These are potential outcomes that These are potential outcomes that are nonmonetary in natureare nonmonetary in nature

Cost effectivenessCost effectiveness

There could be justice issues (i.e., if There could be justice issues (i.e., if offenders who have committed offenders who have committed murder are released because they murder are released because they are low risk)are low risk)

Example of release of offenders who Example of release of offenders who have “aged out” of crimehave “aged out” of crime

Weighing cost benefit and cost Weighing cost benefit and cost effectivesseffectivess

Costs and benefitsCosts and benefits

How effective and what effects, must How effective and what effects, must a program have to be “worth” it?a program have to be “worth” it?

Value judgments Value judgments Political issuesPolitical issues Effects of restricting handgunsEffects of restricting handguns Mandatory jail sentences for abortion Mandatory jail sentences for abortion

protestorsprotestors

Costs and benefitsCosts and benefits

Policies concerning drugs, drug law Policies concerning drugs, drug law sentencessentences

Racial profilingRacial profiling Local example of pornographyLocal example of pornography Does a program work—obviously Does a program work—obviously

there are stakeholders who want it to there are stakeholders who want it to workwork

Many issues in evaluationMany issues in evaluation