Conducting Economic Analysis

7/27/2019 Conducting Economic Analysis

1/33

A MANUAL ON

CONDUCTING ECONOMIC ANALYSIS OF

CRIME PREVENTION PROGRAMS

Prepared for:

National Crime Prevention Centre

Prepared by:

Joseph P. Hornick, Ph.D.

Joanne J. Paetsch, B.A.

and

Lorne D. Bertrand, Ph.D.

Canadian Research Institute for Law and the Family

June 2000


2/33

The views in this manual are those of the authors and do not

necessarily represent the views of the Canadian Research Institutefor Law and the Family or the National Crime Prevention Centre.


3/33

PREFACE

The Canadian Research Institute for Law and the Family (CRILF) was contracted toorganize a consultation for the National Crime Prevention Centre (NCPC) on cost/benefitanalyses of crime prevention programs. The goal of the consultation was to develop a meth-odology for conducting cost effectiveness and cost benefit analyses of NCPC-funded pro-grams. The consultation took place in Ottawa on January 24 and 25, 2000. Eight expertswere invited to attend, as well as representatives from NCPC, the Auditor General, and theTreasury Board (please see Appendix A for a List of Participants). An information packagecontaining recent publications, a discussion paper, and an agenda was circulated to partici-pants prior to the consultation (please see Appendix B for a List of Circulated Publications).

The consultation was facilitated by Dr. Joseph P. Hornick, Executive Director, CRILF,and Ms Patricia Begin, Director, Research and Evaluation, NCPC. Ms Joanne Paetsch,Administrator/Research Associate, CRILF, took notes of the discussions. The consultationbegan with an introduction and general framework by Dr. Hornick, followed by a summary of

NCPC activities and expectations by Ms Begin. Dr. Brandon Welsh then presented an issuepaper, which guided subsequent discussions. This manual is based on the recent publica-tions, the discussion paper and the discussions during the two-day consultation. Publica-tions that are particularly relevant to the manual include:

(1) Dhiri, Sanjay, & Brand, Sam. (1999). Analysis of Costs and Benefits: Guidance forEvaluators. London, England: Home Office Research, Development and StatisticsDirectorate; and

(2) Karoly, Lynn A., Greenwood, Peter W., Everingham, Susan S., Houb, Jill, Kilburn, M.Rebecca, Rydell, C. Peter, Sanders, Matthew, & Chiesa, James. (1998). Investing in

Our Children: What We Know and Dont Know About the Costs and Benefits of EarlyChildhood Interventions. Santa Monica, CA: Rand Corporation.

(3) Welsh, Brandon C. Applying Economic Analysis to Crime Prevention: Issues for aNational Approach. *This paper was prepared for the International Centre for thePrevention of Crime and the National Crime Prevention Centres Consultation onCost-Benefit Analyses of Crime Prevention Programs, Ottawa, Canada, January24-25, 2000.

The authors would like to acknowledge the assistance and cooperation of a number ofindividuals who made it possible to complete this project. Ms Patricia Begin provided guid-ance throughout the project, and co-facilitated the consultation. Ms Linda Bland typed themanual and provided clerical assistance throughout the project. Lastly, the project could nothave been completed without the contributions of the experts who participated in the consul-tation. This project was funded by the National Crime Prevention Centre. The CanadianResearch Institute for Law and the Family is supported by a grant from the Alberta LawFoundation.


4/33

TABLE OF CONTENTS

1.0 Introduction ........................................................................................................... 1

1.1 Background: The National Crime Prevention Centre Strategy ................... 1

1.2 The Crime Prevention Investment Fund ...................................................... 11.3 Objective of this Manual .............................................................................. 21.4 How to use this Manual ............................................................................... 2

2.0 The Process and Outcome Evaluation: The Basis for Cost Benefit Analysis 3

2.1 A Model for Process and Outcome Analysis ............................................... 32.2 Need for Rigorous Evaluation ...................................................................... 62.3 Limitations of Current Projects ..................................................................... 6

3.0 Cost Effectiveness and Cost Benefit Analysis .................................................. 8

3.1 Definitions .................................................................................................... 83.2 Types of Economic Analysis ........................................................................ 93.3 Steps in Economic Analysis ......................................................................... 123.4 Responsibilities ............................................................................................ 14

4.0 Measuring Costs ................................................................................................... 15

4.1 Start-up and Ongoing Costs ........................................................................ 154.2 Specific Categories for Costs ...................................................................... 15

5.0 Measuring Outcomes ........................................................................................... 19

5.1 Identification of Outputs ............................................................................... 195.2 Identification and Quantification of Outcomes ............................................. 19

6.0 Further Tasks: Measuring Benefits .................................................................... 21

6.1 Valuing Outcomes........................................................................................ 216.2 Comparing Costs with Economic Benefits ................................................... 216.3 Other Issues to be Considered ................................................................... 22

References ....................................................................................................................... 24

Appendix A List of ParticipantsAppendix B List of Circulated Publications


5/33

1.0 INTRODUCTION

1.1 Background: The National Crime Prevention Centre Strategy

The Government of Canada launched the National Strategy on Community Safetyand Crime Prevention in June 1998. It is currently in its second year of a five-year fundingcycle. The objectives of this crime prevention strategy are to:

Promote the integrated action of key partners to reduce crime and victimization;

Develop and implement community-based solutions to problems that contribute to

crime and victimization, particularly as they affect children, youth, women, and Ab-original persons; and

Increase public awareness and support for effective approaches to crime prevention.

Key to the National Strategy are: (1) strategic partnerships with other federal govern-ment departments, provincial and territorial governments, municipalities, non-governmentalorganizations, associations, communities, and the private sector; (2) a belief that to be effec-tive, solutions to crime and victimization must be community-based; and (3) a recognitionthat there are limits to the criminal justice systems ability to prevent crime and that enforce-

ment, courts and corrections should be balanced with a social development approach thatattempts to deal with the root causes of crime and victimization. The aim of this initiative is toequip Canadians with the knowledge, skills and resources they need to advance efforts thataddress crime and victimization in their communities.

There are four funding components to the National Strategy: (1) the Community Mo-bilization Program; (2) the Crime Prevention Partnership Program; (3) the Crime PreventionInvestment Fund; and (4) the Business Action Program on Crime Prevention. The programsfunded under the Crime Prevention Investment Fund are the focus of this manual.

1.2 The Crime Prevention Investment Fund

The Investment Fund is the Research and Development component of the NationalStrategy. Its principal objectives are to identify and evaluate as rigorously as possible com-munity-based, social development program approaches to the prevention of crime and vic-timization. The priority is high-need, low-resource communities. The main goal of the Invest-ment Fund is to establish reliable information on what works and what is promising in reduc-ing the risk factors and costs associated with crime and victimization.

1


6/33

Twenty crime prevention demonstration programs are currently being conducted, or areabout to be implemented. They include community-based, social development strate-gies supporting children, families and communities at risk of crime and victimization.Approximately eight more programs are in various stages of development and will beimplemented early in the fiscal year 2000-2001. With few exceptions, the programssupported under the Investment Fund focus on at-risk children and youth and their

families. (Visit NCPC Website athttp://www.crime-prevention.org/english/crime/programs.html for details on the CPIF projects.)

All of the programs funded under the Crime Prevention Investment Fund are beingevaluated using a process and outcome analysis strategy. When possible, evaluators are tobe encouraged to use the same or comparable measures of outcomes with similar targetgroups. To facilitate this, the NCPC has developed a list of strategic crime prevention meas-ures for various types of programs including: family-based initiatives; peer group initiatives;and school-based initiatives. This strategy will facilitate the later comparison of researchresults and cost benefit analysis, thus building a comprehensive knowledge base in this area,

which will support the development of sound policy.

1.3 Objective of this Manual

In addition to process and outcome analysis, it is the goal of this manual to provide amodel for conducting economic analysis. If done properly, this analysis can be used as a toolfor making choices between alternative uses of resources and/or alternative distribution ofservices. Two levels or types of analyses are covered in this manual and reflect the specificobjectives of the manual, which are as follows:

(1) to provide a model for the research evaluators to carry out cost-effectiveness analysisof the programs they are evaluating; and

(2) to provide the basis for the NCPC to carry out (or arrange to carry out) a comparativecost-benefit analysis, based on the cost data and outcomes of the individual pro-grams.

1.4 How to use this Manual

This manual is designed to be a practical tool for evaluators of Crime Prevention Invest-ment Fund projects. It includes a description of the basic tasks and responsibilities for con-ducting a rigorous process and outcome evaluation, which is the basis for cost-benefits analysis(Section 2.0). It defines and describes cost-effectiveness and cost-benefit analyses (Section3.0), and it focuses in some detail on the task of gathering information on costs (Section 4.0).In the next section (Section 5.0), issues related to the identification and measurement ofoutcomes are discussed and finally the last section briefly outlines the future tasks of deter-mining the monetary benefits (Section 6.0).

2
http://www.crime-prevention.org/english/crime/programs.htmlhttp://www.crime-prevention.org/english/crime/programs.htmlhttp://www.crime-prevention.org/english/crime/programs.htmlhttp://www.crime-prevention.org/english/crime/programs.htmlhttp://www.crime-prevention.org/english/crime/programs.htmlhttp://www.crime-prevention.org/english/crime/programs.html


7/33

2.0 THE PROCESS AND OUTCOME EVALUATION:THE BASIS FOR COST BENEFIT ANALYSIS

The critical factor in conducting economic analysis of a group of prevention programs isthe methodological quality of the project-specific research. The economic analysis can only

be as good as the process and outcome research upon which it is based. Weak and flawedresearch will limit the validity of the findings of the economic analysis.

2.1 A Model for Process and Outcome Analysis

Process Analysis

In the broadest terms, process analysis examines how a program or project was actu-ally implemented and answers the question of whether the program was carried out as it wasintended. It monitors and documents the goals, inputs, activities, and outputs as pictured in

Figure 1. A complete process analysis must include a comprehensive coverage of all thedesignated components outlined in Figure 1.

Program Goals and Objectives

The programs goals and objectives must be clearly documented in a process analysis.Further, the objectives must be measurable since they form the basis for expected outcomefor the outcome analysis. At this stage, the complete program design and rationale shouldbe documented, as well as anticipated inputs, activities, and outputs. If these expectationsare changed as the program is implemented, then the reason why this occurred should beclearly documented.

Inputs

Inputs are the next component that must be documented in a process analysis. Theyinclude all the resources used to implement and operate the program. Most inputs can be(and must be for cost analysis) expressed in terms of money. While this may seem straight-forward when considering such things as staff costs, administrative costs, and capital costsnecessary to operate a program, such things as volunteer time, shared administrative struc-tures, etc. are complex and difficult to measure in monetary terms, but must be included todetermine actual costs of a program. The measurement of inputs is critical to cost effective-

ness and cost benefit analysis and will be covered in more detail in Section 4.0.

Activities

The specific activities of the program are also a critical component of a process evalu-ation. Unfortunately, evaluators often assume that program services and activities

3


8/33

Figure 1Process and Outcome Evaluation

Process Analysis

Outcome Analysis

ProgramGoalsObjective

Input Activitie Output Outcome

Programdesign andrational

Resourcesused to theprogram

What theprogramdoes with

resources toaccomplishgoals andobjectives

Direct resultsprograme.g. number of

clients served,needs of clients

Measure ofhow wellprogram ob-

jectives areacheved

4


9/33

are adequately measured by the descriptions in the proposal and do not attempt to measureactivities directly after the program is implemented. In reality, program activities often varyconsiderably from the proposed approach both within and between programs. One of themost accurate measurements of program activities is obtained through the use of a timebudget study. A time budget study tracks for a given period of time the actual activities of the

service providers, which is most representative of the normal operation of the program. Alter-natively, reviews of files, interviews with the service providers, or focus groups can be con-ducted to collect this information, although these methods tend to be less objective.

While a detailed description of the activities is not required for economic analysis, it isessential for at least two reasons. First, it facilitates possible replication of the program.Second, if the activities are not monitored directly, differences in the way similar programsare implemented could result in achievement of different unexplained outcomes.

Outputs

Outputs are the direct results of program activities and are usually measured in terms ofthe amount of work accomplished. For example, they could include the number of staff hired,number of clients served, the level of client service delivered, education/training sessionsheld, and coordinating or partnering with other agencies. If clients are serviced directly, it isalso essential during this step of the process analysis to document their needs and strengthsat intake into the programs since these directly affect potential for change and the final out-comes.

Outputs can best be measured by a combination of approaches including: time budgetstudies, review of client files, and baseline standardized measures of clients needs. It isimportant that outputs not be confused with outcomes. Further, it is the outcomes attribut-able to the intervention that are important for economic analysis.

Outcome Analysis

In addition to the four process analysis components described above, as indicated inFigure 1, an outcome analysis includes measurement of outcomes.

Outcomes

Outcomes are the measures of whether the programs are having their intended ef-

fects by achieving specific program objectives, identified during the design phase of theprogram. In programs involving the provision of services to clients, the outcomes shouldfocus directly on the changes expected of the clients. For example, outcomes may relateto behaviour, knowledge, attitude, values, or other attributes that are affected by the pro-gram. In addition, for some programs (e.g., crime prevention through social development),it may also be necessary to identify shorter-term outcomes since the full benefits of theprogram may not be realized for many years. While intended outcomes

5


10/33

(i.e., those dictated by the stated objectives of the program) are the core of outcome evalua-tion, researchers should also be sensitive to unintended effects both positive and negative.

2.2 Need for Rigorous Evaluation1

The stronger the research design of the outcome evaluation, the more confidence canbe placed in the findings of the economic analysis. Some researchers (e.g., Weimer & Fried-man, 1979) recommend that the cut-off point for performing economic analyses be set atexperimental or strong quasi-experimental designs. Anything less will reduce the confidencethat can be held in the economic findings.

This is a particularly difficult issue, since the use of randomized controlled experi-ments or even the establishment of comparison groups may not be possible for a variety ofreasons, including community or political resistance, a treat all approach of the interven-tion, methodological issues (e.g., small sample size), and financial cost. Simple one-group(no control group) pre-post evaluation designs are better than nothing at all, but attributing

reported claims of program effects, whether desirable, undesirable, or unchanged, to theprogram is confounded by many potential alternative explanations. The addition of a controlor comparison group, preferably one that is equivalent to the treatment group, increasessubstantially the ability to make a valid inference of cause and effect. The equivalent controlgroup design program group compared with comparable control group, measuring pre-post and experimental-control comparisons has been recommended as the minimum re-search design for evaluating crime prevention programs (Sherman et al., 1997, 1998).

Further, as Sherman et al. have indicated, an evaluation cannot be only a description ofthe implementation process or monitoring or auditing the expenditure of funds. Measuringimpact requires claims about cause and effect. The scientific standards for inferring causa-tion have been clearly established in the literature on research design and methods (Cook &Campbell, 1979; Federal Judicial Center, 1981). They include, at minimum, the ability to:

(1) measure the dosage, timing and content of the project or program;(2) gather baseline data prior to the start of the program, if necessary; and(3) gather comparable data from both the program group and appropriate comparison

groups where the program is not operating.

We understand, however, that this might not be possible in the current program ofresearch projects. Just focusing on the establishment of a control group when faced with, for

example, a treat all approach, a number of options are available to the researcher, includ-ing the use of time-lag recruited no-treatment control groups or wait-list control groups inwhich the full sample eventually receives the intervention (see e.g., Webster-Stratton &Hammond, 1997).

2.3 Limitations of Current Projects

There are a number of incompatibilities between the ideal of rigorous social science

6


11/33


12/33

3.0 COST EFFECTIVENESS AND COST BENEFIT ANALYSIS

Crime prevention has been identified as a priority in political and policy settings. How-ever, very little is known about the economic efficiency of crime prevention. This is a particu-larly complex problem in the area of crime prevention through social development because ofthe possible distance in time between the intervention and the actual reduction in the prob-ability of the perpetration of crime, particularly with early intervention programs. The need forempirically-based information in this area, however, dictates that we must do the best jobpossible in establishing the true value of crime prevention through social development.

The key questions in this area are as follows:

What is the cost of a specific crime prevention program?

How do outcomes and the cost of the programs compare?

Which specific program approach is the most cost effective in crime prevention andproduces the greatest net benefits for society?

How should future resources be spent?

In this section, cost effectiveness and cost benefit analyses are defined, the stages ofanalysis are identified, and the responsibilities for specific tasks are listed.

3.1 Definitions

The concepts of inputs, activities, outputs, and outcomes have been defined and dis-cussed above. The concepts of costs and benefits however, also need to be defined.

Costscan be defined as the monetary value of inputs. Some of these we direct andchoose, such as the costs of staff salaries. Others are less direct and more difficult tomeasure, such as the costs of volunteer time. A strategy for identifying all costs willbe discussed in the next section of this manual.

Benefitscan broadly be defined as the value of outcomes to society that are attributedto the program intervention. They are usually expressed in monetary terms, but there

may also be significant benefits that cannot be monetized.

8


13/33

3.2 Types of Economic Analysis

Cost Effectiveness Analysis

Cost effectiveness analysis in the current context of crime prevention estimates thecosts of achieving a unit change in a specific outcome, which is measured by either a reduc-

tion in crime directly and/or an increase in conditions known to be associated with the prob-able reduction in crime (where the reduction in crime is theoretical as opposed to real). Inother words, cost effectiveness analysis will demonstrate whether specific programs havebeen more or less costly in achieving specific outcomes than existing intervention or othernew approaches.

Figure 2 pictures the critical components of cost effectiveness analysis and its relation-ship to process analysis and outcome analysis. As indicated by this figure, cost effective-ness analysis examines inputs in terms of costs relative to outcomes in terms of what isachieved by the intervention. For example, we can use cost effectiveness analysis to deter-mine the unit cost per family of delivering home visitors interventions to boost the healthy

development of infants and school readiness of young children. Further, if outputs and out-comes of several different program interventions are measured in the same way, we can usecost effectiveness analysis to compare the relative cost effectiveness of the different inter-ventions.

The hypothetical examples in Table 1 demonstrate the usefulness of cost effectivenessanalysis. As is noted in the table, Program A is more expensive in terms of unit costs perclient. However, cost effectiveness analysis demonstrates that Program A is far more costeffective than Program B with respect to diversion of children from child welfare.

Cost Benefit Analysis

As shown in Figure 2, cost benefit analysis is based on the input component of processanalysis and outcome analysis. Cost benefit analysis, however, involves a critical additionalstep, i.e., calculating the outcomes of the intervention in monetary terms. Once both inputsand outcomes are expressed in monetary terms, then benefit/cost ratios can be calculatedwhere the value of outcomes is divided by input costs, or the net benefit can be calculated bysubtracting the sum of the input costs from the sum of the benefit costs. While the calcula-tions of cost benefit analysis seem to be straightforward, the assumptions upon which theamounts of monetary benefits are based are quite complex and will be discussed in Section5.0.

The obvious advantage of cost benefit analysis over cost effectiveness analysis is thatit permits the comparison of the cost/benefit of interventions which might have very differentoutcomes and target groups because the outcomes are expressed in common terms, i.e.,money.

9


14/33

Figure 2A Model of Process, Cost Effectiveness, and Cost Benefit Analysis

Process Analysis

Outcome Analysis

ProgramGoalsObjective

Input Activitie Output Outcome

Cost Effectiveness Analysis

Cost Nenefit Analysis

Inputs Outcomes

InputsMonetizeOutcome(Benefits)

10


15/33

Table 1

Hypothetical Example of the Cost-Effectiveness Analysisof Two Early Intervention Programs

PROGRAM A: HOME VISITATION

DataCost of the program for 1 year = $150,000Number of clients served in 1 year = 50Outcome: number of clients diverted from child welfare = 30

AnalysisUnit cost of program per client = $3,000(150,000 ) 50)Average cost of diversion from child welfare = $5,000(150,000 ) 30)

PROGRAM B: PARENTING EDUCATION COURSE

DataCost of the program for 1 year = $100,000Number of clients served in 1 year = 100Outcome: number of clients diverted from child welfare = 10

AnalysisUnit cost of program per client = $1,000(100,000 ) 100)Average cost of diversion from child welfare = $10,000(100,000 ) 10)

COMPARISON

Unit cost of Program A is three times that of Program B.But Program A is twice as cost-effective than Program B in diverting

clients from child welfare.

11


16/33

The hypothetical example in Table 2 (based on the example in Table 1) demonstratesthe increased usefulness of cost benefit analysis over cost effectiveness analysis. Note thatbecause of the relative high cost of child welfare involvement, Program A obtained a highbenefit to cost ratio, i.e., 1.6, and the overall monetary benefit of the program outweighed thecost by $90,000. Program B in contrast cost less but was less effective and thus generateda cost of $20,000 rather than any economic benefit.

In terms of crime prevention interventions, many outcomes will be measured by eitherconditions associated with the reduction of crime, or the actual reduction of crime. Thus,benefits are usually based on the reduction of costs (i.e., savings) to society of the costs ofcrime including costs to the victim(s) and the justice system. In order to calculate the savingsto society of a particular intervention, how much crime has been prevented as a result of theintervention must be identified.

3.3 Steps in Economic Analysis

In review of the process pictured in Figure 2, cost effectiveness analysis involves thefollowing steps:

1. Describing the intervention in terms of its goals and objectives and activities.

2. Identifying and quantifying inputs.

3. Valuing inputs in money.

4. Identifying outputs.

5. Identifying outcomes.

6. Quantifying outcomes.

7. Comparing input costs with outcomes.

Cost benefit analysis adds two other steps:

8. Valuing outcomes thus creating benefits.

9. Comparing economic costs with economic benefits.

12


17/33

Table 2

Hypothetical Example of the Cost-Benefit Analysisof Two Early Intervention Programs1

PROGRAM A: HOME VISITATION

DataInput cost = $150,000Outcome quantity (diversion from child welfare) = 30Outcome value (benefit)2 = $240,000(30 x 8,000)

AnalysisBenefit/cost ratio = 1.6

(240,000/150,000)Net economic benefit = $90,000(240,000 - 150,000)

PROGRAM B: PARENTING EDUCATION COURSE

DataInput cost = $100,000Outcome quantity (diversion of child welfare) = 10Outcome value (benefit)2 = $80,000

(10 x 8,000)

AnalysisBenefit/cost ratio = .8(80,000/100,000)Net economic benefit (cost) = -$20,000(80,000 - 100,000)

COMPARISON

For Program A, benefits outweigh costs by $90,000.For Program B, costs outweigh benefits, and there is a net cost of$20,000.

1 Using the example from Table 1.2 Average hypothetical value of a diversion from child welfare = $8,000.

13


18/33

3.4 Responsibilities

It is expected that the evaluators of specific Crime Prevention Investment Fund projectswould conduct cost effectiveness analysis by carrying out steps 1 through 7 above payingparticular attention to the collection of detailed costs data and the identification and quantifi-cation of outcomes. This cost effectiveness analysis then would facilitate direct comparisons

between different interventions with similar outcomes. In addition, the evaluators of specificprograms would report detailed data on inputs and outcomes to the NCPC in order to facili-tate both comparative cost effectiveness analysis and cost benefit analysis.

The NCPC then would be responsible for the completion of the cost benefit analysis(i.e., tasks 8 and 9) either directly or through contracting specifically for these tasks. It isimportant that these tasks are centralized in the NCPC to facilitate standardization in thecomparison of the benefits of different programs.

14


19/33

4.0 MEASURING COSTS

In order to complete steps 2 and 3 of the economic analysis (see Section 3.3), inputsinclude an array of resources which must be identified, quantified, and expressed in mon-etary terms to measure the cost of an intervention. These resources should be identified as

soon as possible by the program evaluators, so data can be collected on an ongoing basisrather than retrospectively. If the program is already operating, retrospective data will besufficient; however, extra effort may be required to collect comprehensive cost data.

4.1 Start-up and Ongoing Costs

Obviously the start-up or initiation of a program could cost considerably more than on-going operations. The hiring and training of new staff combined with the gradual intake ofclients make this stage of a program less cost effective than it would be once the staffing hasstabilized and the case load has reached maximum numbers. However, since we are cur-

rently interested in the possible replication of these crime prevention programs, identificationof start-up costs as well as ongoing operations is essential. Presumably the first year ofoperation of these programs would be a reasonable time period to identify start-up costs.After the first year of operation, it is hopeful that the programs would be fully operational. Ifthis is not the case, it would seem to be a good reason to examine implementation issuessuch as the need for the intervention, intake procedures, or advertising, and adjust the pro-gram to fit the revised need. Once the program has reached a full complement (e.g., caseloadis at full capacity) of service provisions, measuring ongoing costs should commence.

Because we are currently concerned with both replicating and sustaining these pro-grams, both start-up and ongoing costs should be identified by performing cost effectivenessanalysis on the data for the first and second year of operation if the program is fully opera-tional. It should be recognized that programs which require community capacity building andthe formation of partnerships in order to get them off the ground may take a number of yearsbefore the program is fully operational, and ongoing costs can be identified.

4.2 Specific Categories for Costs

Table 3 contains a list of direct and indirect costs of a typical program. Direct costsshould be relatively easy to identify since they are usually already identified in monetary

terms. Indirect costs, such as volunteer time, however, are more difficult to identify but mustbe included in the cost analysis.

15


20/33

Table 3

Cost Data Required for Crime Prevention Interventions

A: DIRECT COSTS

Administrative costs staff salaries and benefits staff training costs (start-up and in-service)

Operating expenses equipment (purchase and rental) furniture office space (if purchased or rented) office supplies utilities (heat, water, electricity) telephone, fax, mail, etc. advertising insurance (e.g., liability, automotive, property) transportation

Client expenses materials for clients (e.g., books on parenting) travel expenses (e.g., bus fare for appointments, mileage for special outings, child

care costs)

B: INDIRECT COSTS

Volunteer (time) for partner networks and administration

Volunteer (time) for client services

Victim costs

In-kind resources (e.g., resources shared from host agency such as space, furniture, com-

puters)

16


21/33

Direct Costs

Direct costs include all the expected cost items that would be included in a pro-gram budget. Evaluators should begin to monitor these costs as early as possible to ensurethe calculation of start-up costs as well as ongoing costs. As Table 3 indicates, direct costsinclude at least three categories of costs. First, administrative costs include such items as

staff salaries and benefits. Pay particular attention when positions are split within an agencyto include the correct proportional costs of the position to the program being evaluated. Stafftraining is also a significant administration cost, particularly at the start-up of new programs.Even after programs are up and running, in-service training can be a significant expenditureand should be carefully documented. In-service training should be interpreted broadly toinclude a full range of educational activities, e.g., from attendance at conferences to specialtargeted programs such as a management or supervision course.

The next major category of direct costs is capital costs. It includes both purchaseand rental of all the equipment necessary to operate the program. Office space is anotheroperating cost that can be simple, e.g., the cost of rental of premises or it can involve the

purchase cost of a building. Office space may also be donated by a host agency, in whichcase it should be listed under indirect costs as use of an in-kind resource. All equipment suchas computers, fax machines, and copy machines should be itemized here and it should beindicated whether they are rented or purchased. Likewise, if the program owns a vehicle, itshould be itemized here.

Other operating expenses should also be tracked by item. These include such basicsas stationery, mail, and courier costs, in addition to utilities and telephones. Advertising couldinclude the costs of a range of things such as brochures to special events staged with theintent of advertising the program. Insurance can also be a considerable cost. All insurancecosts should be recorded such as property, liability for staff and boards of directors, andautomobile insurance if the program owns a vehicle. Transportation costs should also becarefully tracked. This would include a range of things from reimbursing staff mileage whenthey have to travel for work purposes to travel to administrative meetings, courses, and/orconferences.

Some direct expenses may also be attributed to clients. Some programs, for example,have to purchase material such as books, training and/or assessment materials for clients.Travel for clients to meetings or special outings as well as child care costs may be coveredand thus should be documented.

Indirect Costs

As mentioned above, indirect costs should also be tracked, but these are often far moredifficult to measure than direct costs. Indirect costs can be categorized into at least threecategories.

The use of volunteers is probably the most common indirect expense and it occurs attwo levels administrative and client service and both of these should be

17


22/33

measured. Administrative volunteers would include the volunteer who sits on an advisoryand/or administrative committee for the program. Their time is either totally voluntary (i.e.,they are not paid) or contributed (i.e., they are paid to sit on the committee by a partneringagency). Regardless, in all cases volunteer time should be identified, and assessed a mon-etary equivalent.

In cases where a volunteers time is paid for by another agency and as such the volun-teer is acting as an agent or representative of another agency or program, that volunteer timeshould be recorded (or at least estimated if precise records are not available) and the equiva-lent of the volunteers hourly wage should be used to estimate the indirect costs. In situationswhere volunteers are operating independently on their own time, then the estimate of theircosts should be based on the level of tasks that they are performing for the program.

Cohen (1998) has also indicated that the cost of crime to victims should, if possible, beestimated. This may include direct financial loss as well as the indirect cost of pain andsuffering. Unfortunately, it is difficult to monetize the indirect cost. However, at minimum,they should be mentioned.

In-kind resources are particularly important to consider in programs that are hosted bylarger agencies that also have many other programs. The resources can range as indicatedabove from free office space to the use of agency computers and information systems. Allof these in-kind resources should be identified and monetary estimates of costs should bemade in order to determine the true costs of implementing a program.

18


23/33

5.0 MEASURING OUTCOMES

This section deals with the completion of steps 4 through 7 of the economic analysislisted in Section 3.3 (i.e., identification of outputs and outcomes; quantification of outcomes;and comparison of costs with outcomes). The completion of these tasks, in addition to steps

2 and 3 (measuring costs), provides the framework for the cost effectiveness analysis de-scribed in Section 2.0.

5.1 Identification of Outputs

For cost effectiveness analysis, it is necessary to identify both the number of clientsserved and the level of service delivered. When this information is analyzed in relation to thecosts of inputs, it provides the basic unit cost per client and cost of specific services deliv-ered.

5.2 Identification and Quantification of Outcomes

As discussed previously, outcomes are the condition and/or changes in conditions thatresult because of the activities of the program. The identification and quantification of out-comes is a complex task due to a number of issues including intended and unintended out-comes, short-term and long-term outcomes, and qualitative outcomes.

Intended and Unintended Outcomes

Intended outcomes include the outcomes derived from specific program objectives. Inother words, the intended outcomes for programs involved in the provision of services toclients should measure directly the changes expected of the client as a result of the interven-tion. These may relate to behaviour, knowledge, attitude, or values that are expected tochange because of the program. Intended outcomes measure whether the program is achiev-ing the impact that it is intended to provide.

Unintended outcomes are more difficult to identify and measure because they are nottied to program objectives and are not necessarily anticipated. Further, they can be positiveor negative. An example of a negative unintended outcome was the increase in theft ofcigarettes that occurred a few years ago. Governments increased tax to get people to stop

smoking; however, the unintended effect was that it made cigarettes so expensive that morepeople were willing to take the risk of stealing them.

While it is difficult if not impossible to identify unintended outcomes when an evalua-tion study is first implemented, researchers should remain vigilant and open-minded inmonitoring the effects of programs. If there is a suspicion that unintended

19


24/33

outcomes are occurring, if possible a strategy should be developed to measure these out-comes.

Short-term/Long-term Outcomes

The distinction between short-term and long-term outcomes is particularly relevant for

crime prevention programs. Obviously the ultimate goal of all crime prevention programs isthe reduction in crime. For some crime prevention initiatives, particularly target hardening(e.g., increased security), the reduction of crime may be relatively direct and immediate.However, in the area of crime prevention through social development, observable reductionin crime is not so direct or immediate. For example, interventions focussed on the develop-ment of young children and families create particular problems for the measurement of im-pacts and outcomes since their main benefits are only expected to occur in the medium tolong term. The full benefits of such a program may not be visible for many years and thesuccess of the intervention will need to be measured in the interim period. Intermediateimpact measures should therefore be established that partly reflect final outcome measuresand partly assess any benefits that occur in the short term (Karoly et al, 1998).

At minimum, program effects on delinquency and later offending should be measured.An assessment of crime benefits should focus on affected agencies within the criminal jus-tice system (e.g., police, courts, corrections) and crime victims and their families. Other keyoutcome variables that should be measured include substance abuse, education, employ-ment, health, and family factors. In the case of education, for example, benefits should beassessed for schooling expenses (e.g., remedial classes, support services) and educationaloutput (e.g., high school completion, enrollment in college or university). Other outcomevariables such as conduct problems in early childhood should also be measured (Welsh,draft).

Qualitative Outcomes

Outcomes which cannot be quantified should also be documented even though theycannot be used in the cost-effectiveness or cost benefit analysis. For example, a police footpatrol program, which increases police visibility, may lead to a reduction in fear of crime evenif it does not affect the crime rates directly.

Comparison of Costs with Outcomes

Once outcomes have been identified and measured they can be compared with the

costs of the activities performed to achieve them, thus completing cost effectiveness analysis(see Section 3.2).

20


25/33

6.0 FURTHER TASKS: MEASURING BENEFITS

The final two steps in economic analysis involve valuing outcomes thereby creatingbenefits, and comparing costs with economic benefits. These tasks should be performedcentrally under the direction of the NCPC since all projects should be measured by the same

standards. These tasks go beyond the mandate of this manual; however, a brief discussionof them is presented below.

6.1 Valuing Outcomes

Valuing outcomes of crime prevention initiatives is a complex process that requires atminimum the identification of standardized estimates of the costs of crimes and the identifi-cation of how many potential crimes (or crime careers) were prevented by a particular pro-gram which achieved specific outcomes.

Estimates of the Costs of Crime

Estimates of the costs of specific crimes will need to be calculated on the basis ofexisting data on the rates of crime and the costs associated with dealing with those crimes.These estimates of the unit costs of different types of crimes will allow the valuation of crimesprevented. The use of uniform unit costs of crimes is necessary in comparing different crimeprevention programs.

Estimating Crimes Prevented

Estimating crimes prevented must be based on a theory that links the outcomes ofprograms to the estimated number of crimes prevented by that program. In the case of somevery targeted programs, such as an initiative focussed on reducing re-offending among first-time young offenders, the link can be relatively direct and easy to calculate. In other initia-tives, where the link is more theoretical and longer term, the prediction will be more difficult e.g., programs which focus on the social development of young children.

6.2 Comparing Costs with Economic Benefits

Once the outcomes of a crime prevention initiative are valued, e.g., an estimate can be

made of the number of crimes prevented by the program, as well as the cost savings ofpreventing the crimes, then a comparison can be made with the costs of the program.

The economic efficiency of a program can be reported as a benefit-cost ratio by dividingthe value of the outcomes (or benefits) by the costs of the program. A benefit-cost ratiogreater than 1.0 means the program is economically efficient.

21


26/33

The economic efficiency of a program can also be reported as a net value, by subtract-ing the costs of the program from the benefits, or value of the outcomes. A positive numbermeans the program is economically efficient.

Other Cost Savings

While the generally accepted formula for identifying the cost/benefit of a crime preven-tion program is briefly outlined above, other cost savings should also be considered espe-cially since many of the NCPC initiatives have a broad focus on crime prevention throughsocial development. In other words, many of these programs may do more than preventcrime and there should be some attempt to calculate these benefits.

Karoly et. al. (1998) provide an excellent example of this in their analysis of early child-hood intervention. As they indicate, early childhood intervention programs generate at leastfour types of cost savings:

(1) increased tax revenues from the increased employment earnings of program partici-

pants;

(2) decreased welfare outlay due to higher employment;

(3) reduced expenditures for health services; and

(4) lower criminal justice system costs.

6.3 Other Issues to be Considered

There are a number of other issues that should be considered by a centralized teamtasked with conducting cost benefit analysis of crime prevention programs. These includethe following:

Calculating present value Since many of the projects funded by NCPC will involveprograms lasting more than one year, it is necessary to take inflation into account.This can be done by calculating present value, i.e., making all monetary costs andbenefits comparable over time by use of a price index.

Marginal costs versus average costs According to Dhiri & Brand (1999), marginal

cost refers to the extra spending that is required to achieve one extra unit of outcome,i.e., to prevent one more crime. It does not include fixed costs, unless they are re-quired to achieve the outcome. If the cost of a marginal increase in spending resultsin a greater reduction in the cost of crime, then, in theory, it is worth increasing spend-ing for the program. Average cost refers to the total costs divided by the outcomes,which allows a comparison of the cost effectiveness of different programs.

22


27/33

Sensitivity analysis A sensitivity analysis should be conducted to check the validityand test the effects of variations in the assumptions made in the economic analysis(Welsh, draft). For example, the assumption of how much of the outcome can beattributed to the program can be varied to provide a range of estimates of outcomesand benefits.

Replicability Economic analysis should include an assessment of the replicability ofthe program. This assessment should specify the conditions under which the programwould be expected to be cost effective. It should also include an indication of whetheran expanded program would be as cost effective as the original program.

23


28/33

REFERENCES

Dhiri, Sanjay, & Brand, Sam. (1999). Analysis of Costs and Benefits: Guidance for Evalua-tors. London, England: Home Office Research, Development and Statistics Directo-

rate.

Karoly, Lynn A., Greenwood, Peter W., Everingham, Susan S., Houb, Jill, Kilburn, M. Rebecca,Rydell, C. Peter, Sanders, Matthew, & Chiesa, James. (1998). Investing in Our Chil-dren: What We Know and Dont Know About the Costs and Benefits of Early Child-hood Interventions. Santa Monica, CA: Rand Corporation.

Sherman, L.W., Gottfredson, D.C., MacKenzie, D.L., Eck, J.E., Reuter, P., & Bushway, S.D.(1998). Preventing Crime: What Works, What Doesnt, Whats Promising. Researchin Brief, July. Washington, DC: National Institute of Justice, US Department of Jus-tice.

Sherman, L.W., Gottfredson, D.C., MacKenzie, D.L., Eck, J.E., Reuter, P., & Bushway, S.D.(1997). Preventing Crime: What Works, What Doesnt, Whats Promising. Washing-ton, DC: National Institute of Justice, US Department of Justice.

Webster-Stratton, C., & Hammond, M. (1997). Treating Children with Early-onset ConductProblems: A Comparison of Child and Parent Training Interventions. Journal of Con-sulting and Clinical Psychology, 65, 93-109.

Weimer, D.L., & Friedman, L.S. (1979). Efficiency Considerations in Criminal Rehabilitation

Research: Costs and Consequences. In L. Sechrest, S.O., White, & E.D. Brown(Eds.), The Rehabilitation of Criminal Offenders: Problems and Prospects, pp. 251-272. Washington, DC: National Academy of Sciences.

Welsh, B.C. (draft). Applying Economic Analysis to Crime Prevention: Issues for a NationalApproach. Paper prepared on behalf of the International Centre for the Prevention ofCrime for the National Crime Prevention Centres Consultation on Cost/Benefit Analy-ses of Crime Prevention Programs, Ottawa, January 24-25, 2000.

24


29/33

APPENDIX A

LIST OF PARTICIPANTS

25


30/33

CONSULTATION ON COST/BENEFIT ANALYSES OFCRIME PRVENTION PROGRAMS

Consultants

Dr. Peter Greenwood

Senior ScholarRAND Criminal Justice Program

and Director, RAND Cyber Assurance1700 Main StreetSanta Monica, CA 90407USATel: (310) 393-0411, Ext.: 6321Fax: (310) 451-7004e-mail: [email protected]

Dr. Ross HastingsDepartment of CriminologyUniversity of Ottawa25 University StreetP.O. Box 450, Station AOttawa, ON K1N 6N5Tel: (613) 562-5800, Ext.: 1797Fax: (613) 562-5304

Mr. Richard Kerr870 Walker Hook RoadSalt Spring Island, BC V8K 1B6Tel: (250) 537-9432Fax: (250) 537-5645e-mail: [email protected]

Mr. Michel LaurendeauSenior AnalystTreasury Board of Canada, SecretariatProgram Evaluation and ReviewLEsplanade LaurierFloor 09WO, 300 LaurierOttawa, ON K1A 0R5Tel: (613) 952-3112Fax: (613) 952-3247e-mail: [email protected]

Dr. Werner J. Mller-ClemmAuditor & Functional

Responsibility LeaderOffice of the Auditor General of Canada240 Sparks StreetMail Stop 10-13Ottawa, ON K1A 0G6Tel: (613) 995-3708, Ext.: 5014Fax: (613) 947-6886e-mail: [email protected]

Dr. Daniel SansfaonInternational Centre for

Prevention of Crime#2100, 507 Place dArmesMontreal, Quebec H2Y 2W8Tel: (514) 288-6731, Ext.: 225Fax: (514) 288-8763Dr. Carol LaPrairie

Senior ResearcherNational Crime Prevention Centre

Department of Justice Canada284 Wellington StreetOttawa, ON K1A 0H8

26
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]


31/33

Ms Sharon StroickManager, Family NetworkCanadian Policy Research Networks250 Albert StreetSuite 600Ottawa, ON K1P 6M1

Tel: (613) 688-5522Fax: (613) 567-7640e-mail: [email protected]

Ms Sherri TorjmanVice PresidentCaledon Institute of Social Policy1600 Scott StreetSuite 620Ottawa, ON K1Y 4N7

Tel: (613) 729-3340Fax: (613) 729-3896

Dr. Brandon WelshDepartment of Criminal JusticeUniversity of Massachusetts Lowell870 Broadway Street, Suite 2Lowell, MA 01854-3044USATel: (978) 934-4109Fax: (978) 934-3077e-mail: [email protected]

Facilitators

Ms Patricia BginDirector, Research & EvaluationNational Crime Prevention CentreDepartment of Justice Canada

284 Wellington StreetOttawa, ON K1A 0H8Tel: (613) 957-4647Fax: (613) 952-3515e-mail: [email protected]

Dr. Joseph P. HornickExecutive DirectorCanadian Research Institute for Law

and the Familyc/o Faculty of Law

University of Calgary2500 University Drive NWCalgary, AB T2N 1N4Tel: (403) 220-6653Fax: (403) 289-4887e-mail: [email protected]

Recorder

Ms Joanne PaetschAdministrator/Research AssociateCanadian Research Institute for Law

and the Familyc/o Faculty of Law

Observers

Mark Irving and Lucie LeonardResearch Evaluation UnitNational Crime Prevention CentreDepartment of Justice Canada

Susan KellyDepartment of Justice Canada

27
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]


32/33

APPENDIX B

LIST OF CIRCULATED PUBLICATIONS

28


33/33

CONSULTATION ON COST/BENEFIT ANALYSESOF CRIME PREVENTION PROGRAMS

Besserer, Sandra, & Tufts, Jennifer. (1999). Justice Spending in Canada. Juristat(CanadianCentre for Justice Statistics), 19(12).

Brantingham, Paul, & Easton, Stephen T. (1998). The Costs of Crime: Who Pays and HowMuch? 1998 Update. Critical Issues Bulletin. Toronto, ON: The Fraser Institute.

Cohen, Mark A. (1998). The Monetary Value of Saving a High-Risk Youth. Journal of Quan-titative Criminology, 14(1), 5-33.

Dhiri, Sanjay, & Brand, Sam. (1999). Analysis of Costs and Benefits: Guidance for Evalua-tors. London, England: Home Office Research, Development and Statistics Directo-rate.

Greenwood, Peter W. (1998). Investing in Prisons or Prevention: The State Policy MakersDilemma. Crime & Delinquency, 44(1), 136-142.

Greenwood, Peter W., Model, Karyn L., Rydell, C. Peter, & Chiesa, James. (1998). DivertingChildren from a Life of Crime: Measuring Costs and Benefits. Santa Monica, CA:Rand Corporation.

Greenwood, Peter, Model, Karyn L., Rydell, C. Peter, & Chiesa, James. (1996). The Eco-nomic Benefits of Diverting Children from Crime. Challenge, September-October, 42-44.

Harries, Richard. (1999). The Cost of Criminal Justice. Research Findings No. 103. Lon-don, England: Home Office Research, Development and Statistics Directorate.

Karoly, Lynn A., Greenwood, Peter W., Everingham, Susan S., Houb, Jill, Kilburn, M. Rebecca,Rydell, C. Peter, Sanders, Matthew, & Chiesa, James. (1998). Investing in Our Chil-dren: What We Know and Dont Know About the Costs and Benefits of Early Child-hood Interventions. Santa Monica, CA: Rand Corporation.

Welsh, Brandon C. (draft). Applying Economic Analysis to Crime Prevention: Issues for aNational Approach. Paper prepared on behalf of the International Centre for the Pre-

vention of Crime for the National Crime Prevention Centres Consultation on Cost/Benefit Analyses of Crime Prevention Programs, Ottawa, January 24-25, 2000.

Documents

Conducting Economic Analysis