Costs, Cost-Effectiveness and Cost-Benefit Analysis

Brooks BowdenCenter for Benefit-Cost Studies of Education

Teachers College, Columbia University

Clive BelfieldQueens College, City University of New York

IES PI Meeting December 11 2015

Outline

① Foundations of CA+CEA+CBA = “EA”② Need for EA③ Case studies④ Future of EA

Context: • EA research at CBCSE.org• EA teaching: IES Methods Training Grant

(2014-17, #R305B140003)

Case Studies: EA

Socio-Emotional Learning

interventions

Talent Search

Reading Partners

City Connects

Early literacy

4

Case Studies: Displacement

• Outcome specification

Socio-Emotional Learning

interventions• Program fidelityTalent Search

• Treatment contrastReading Partners

• Service mediationCity Connects

• Power of the testEarly literacy

5

COST, COST-EFFECTIVENESS, AND COST-BENEFIT ANALYSIS

FOUNDATIONS

4

Basic Ideas

• Money is scarce; must consider if program is affordable

① Costs Analysis (CA): a basic description of how much resource is needed for a program

• Cannot just look at if a program works; must also look at resources needed for it to work

② Cost-Effectiveness Analysis (CEA): the link between resources needed for programs and their outcomes

• Must look at the economic consequences of a program

③ Cost-Benefit Analysis (CBA): test whether economic returns justify the cost

5

Costs Analysis

• See if program can be afforded

• Enumerate all resources to implement a policy

• Incremental costs over “next best alternative” or “business-as-usual”

• This is an actual research project using ‘ingredients’ method:o Samplingo Quantities of ingredientso Prices of ingredientso Testing for difference in costs

• Average Cost per participant

• AC must be within budget

6

Cost Example

High school mentoring program for 500 students:

4 teachers @ $100,000 per

4 classrooms @ $25,000 per

After-school program for 1,000 students:

6 counselors @ $80,000 per

1 gym @ $120,000 per

Mentoring Program

After-school

Program

Total number of students

500 1,000

Personnel $400,000 $480,000

Facilities $100,000 $120,000

Total Cost $500,000 $600,000

Average Cost $1,000 $600

7

Cost-Effectiveness Analysis

• Find cheapest way to do the thing we have decided to do

• Compares policy alternatives, based on ratio of their costs to a quantifiable (but not monetized) effectiveness measure

• Compare alternative policies, not rule on single policy

• Compare similar policies: scale, target group, expected effects

• CE ratio = $Cost/Unit of Effectiveness

• Lowest CE ratio is most cost-effective

10

CEA Example

HS Dropout rate = 20%

School size = 500

Mentoring program:o Costs $1,000 per studento Reduces dropout rate by 5%

After-school program:o Costs $600 per studento Reduces dropout rate by 2%

Mentoring

program

After-school

program

Total cost $500,000 $300,000

Baseline dropouts 100 100

Yield of new high school graduates

5 2

CE ratio $100,000 $150,000

9

Cost-Benefit Analysis

• See if it is worth spending money on a program

• Compare policy alternatives, based on ratio of their costs to quantifiable and monetized effectiveness measures

• Monetized effectiveness measures are “benefits” (e.g. earnings gains, lower special education spending)

• BC ratio = $Benefits/Cost or B−C = Benefit-Costs

• Worth it if BC ratio > 1 or B−C > 0

12

CBA Example

Mentoring program reduces dropout rate by 5%

Each new high school graduate compared to a dropout over lifetime: o Earns $120,000 more

o Saves taxpayer $30,000 in spending on crime

Mentoring program

Total cost over baseline C $500,000

Yield of new high school graduates 5

Total earnings gain B $600,000

Total crime saving B $150,000

Net Benefit (B-C) $250,000

Benefit-Cost Ratio (B/C) 1.50

13

COST, COST-EFFECTIVENESS, AND COST-BENEFIT ANALYSIS

NEED FOR EA

12

Need for EA in Education

• Many interventions, reforms, programs, policies

• Intensively investigate “What Works”

• Very little guidance on

“..At What Cost?”

“..Is it Worth the Cost?”

13

Examples E

Effects

Class size reduction ✓✓

Reading Recovery ✓✓

High VA teachers ✓✓

Vouchers ✓

Web-learning =

16

Examples E C

Effects Costs

Class size reduction

✓✓ Expensive

Reading Recovery ✓✓ Very expensive

High VA teachers ✓✓ No idea

Vouchers ✓ Cheaper (?)

Web-learning = Cheaper (?)

17

Examples E C E?

Cost-Effective or Efficient?

Class size reduction

Perhaps

Compared to what?

From whose perspective?

Is it affordable?

Reading Recovery

High VA teachers

Vouchers

Web-learning

18

EA Methods

• They are methods:o CA: ingredients methodo CEA: CA + evaluation impactso CBA: CA + shadow pricing methods

• There are methodological standards

• But not like impact evaluations because hypothesis / statistical testing is not determinative

19

EA Practice

• EA is becoming more common

• Slowly and from a low base

• Assumptions rarely harmonized

• Research questions rarely related to each other

• Limited hypothesis testing

20

Value of EA

Conventional answer: • It will help decision-making• If it does not help us decide between X and Y,

it is not needed

New answer: • It will improve education research

generally

19

SESSION II

APPLICATIONS & RECOMMENDATIOS

21

Case Studies

• Outcome specification

Socio-Emotional Learning

interventions• Program fidelityTalent Search

• Treatment contrastReading Partners

• Service mediationCity Connects

• Power of the testEarly literacy

23

OUTCOME SPECIFICATION

CASE STUDY: SOCIO-EMOTIONAL LEARNING

21

SEL Interventions

• Two full reviews of interventions that enhance SE skills (Durlak et al., 2011; Sklad et al., 2012)

• Focus on SE skills or test scores? (Can’t do both)

• SE skills are broad-based, not task-specific

• Need to look at broad set of outcomes• Need to weigh those outcomes• This implies CBA or CBA framework25

The Economic Value of Social and Emotional Learning

26

Outcomes of SEL interventions?

Across the literature outcomes were:• Inconsistent• More or less• Student, teacher, classroom, school

level• Durable or ephemeral• Fast-acting or delayed

24

25

Conclusions

• Better “benefit maps”

• Focus on school-level or class-level effects of SEL

• Examine how fast-acting and durable interventions are

• Generally, pre-specify outcomes (Social Impact Bonds)

29

Clincher

• We propose a cost-effectiveness analysis

• Question always asked is:

How are we measuring effectiveness?

• We don’t know – we are economists

30

PROGRAM FIDELITY

CASE STUDY: TALENT SEARCH

38

• Helps low-income and first-generation college students complete high school and gain access to college

• Many services: academic achievement, access to financial aid, test taking and study skills assistance, academic advising, tutoring, career development, campus visits

Talent Search

32

Program Flexibility

Site-specific variation:• Sites can serve students in G6-12 or cohorts

from 6G through 12G; at few or many schools

• Sites are at schools and at colleges• Programs vary with respect to services

Cost analysis needed to:① Describe the program② Estimate in-kind resources③ Identify most cost-effective sites33

EA of Talent Search

• Cost Analysis: Collected cost data using ingredients method from 6 TX sites and 3 FL sites

• CEA: Linked cost data to effects from Constantine et al. (2006)

• CBA: Shadow priced estimates of earnings gains from high school completion and college attendance

34

35

36

PER YEAR

FL3

TX4

TX3

FL1

TX2

FL2

TX6

TX4

TX1

$640

$2,730

$5,190

Cost per Student per Site(PV at age 18 2010$)

37

FL1

FL3

FL2

TX2

TX4

TX1

TX3

TX6

TX5

$- $30,000 $60,000 $90,000 $120,000 $150,000

$10,830

$148,090

Infinity

Cost-effectiveness Ratios: Cost per HS Completer

Read 180

• For struggling readers G1-12 to improve decoding, fluency, and comprehension skills

• Prescriptive program:o 90-minutes daily of whole-group, small-group and

individualized instruction + videos + feedbacko Class size 15

• Collected cost data using ingredients method from three sites (Levin et al., 2007)

38

Read 180 Cost per Student

Site 1 Site 2 Site 3

Students served 6,701 1,080 2,400

Personnel (teachers) $320 $950 $70

Personnel (administrators, technicians, coordinators)

$50 $400 $60

Equipment/materials(computers, licenses) $250 $150 $140

Other (prof. dev., sub teachers, other) $- $10 $10

Average Cost $610 $1,510 $280 46

JOBSTART

• Education/training program for low-income youth, including remedial education, GED preparation, and job placement assistance

• Youth can remain in the program for up to two years

• MIS data from Cave et al. (2003)

40

41

Corpus Christi

EGOS Denver

San Jose

Chicago

LA Jobs Corps

NY City

$4,470

$4,710

$6,650

$14,760

$16,190

$20,800

JOBSTART Cost per Youth per Site (PV at 18, 2014$)

Mean = $14,900SD = $13,400N =5050

Job Corps: Line Item Costs

Cost per Mean SD Max

Counseling $4,620 4360 $50,020

Admin $3,640 3410 $53,540

Voc. training $2,930 2760 $33,540

Allowance $2,200 1940 $11,110

Child care $1,810 1670 $17,450

Outreach $1,630 1440 $9,860

Education $1,600 1460 $15,210

Med/dental $1,270 1180 $13,190

Food $1,050 1020 $12,280

Conclusions

Cost Analysis yields information on:

• What the program is

• Actual program implementation

• Fidelity• Many evaluations include (idiosyncratic?)

investigation of program fidelity

• Base fidelity on resource use indicators

44

TREATMENT CONTRAST

CASE STUDY: READING PARTNERS

28

Reading Partners

• Supplemental pull-out reading

program:o One-on-one tutoring (2x45 minutes per week)o Dedicated school space and materialso Structured and individualized curriculumo Data-driven instructiono Rigorous ongoing trainingo Instructional supervision and support

• For struggling students in K-5

• Split responsibility:o Run with volunteers supported by AmeriCorps Site

Coordinatorso Site Coordinators supervised weekly by Program

Managers

46

Mobilizing Volunteer Tutors

47

www.mdrc.org/sites/default/files/ReadingPartners_2015_FR.pdf

Evaluating RP

• Multi-site randomized design: o 1,250 students randomly assigned at 19 school siteso Grades 2-5, eligible for Reading Partnerso Evaluated during school year 2012-13o Baseline equivalence of treatment and control

• Cost analysis was part of evaluation:o What resources were needed to implement RP as

per the evaluation?o What proportion of RP costs were borne by the

school?o How did these resource requirements compare with

the resources required to implement other supplemental reading services in these schools?48

RP Context

• Cost ≠ Price

• School selects RP as one option for supplemental reading services

• Children in need must be served – impact relative to other supplemental services (not no intervention)

• Cost Study Sample: 6 of 19 study siteso Chosen based on strong implementation, geographic diversity, and ability to collect

reliable data

• Data collected via implementation observations, teacher surveys, and interviews with RP staff, teachers, reading support staff, and school personnel

• Total social cost calculated by summing the costs of all the resources utilized to implement program (Levin & McEwan, 2001)

o Included both in-kind and financial costs

o Used national prices

o 2013 $USD

• Same approach used to assess the cost of the other supplemental reading programs offered at these six schools49

RP Costs

• Total social cost calculated by summing costs of all resources utilized to implement program (Levin & McEwan, 2001)o Included both in-kind and financial costso Used national priceso 2013 $USD

• Same approach used to assess the cost of the other supplemental reading programs offered at these six schools

50

RP Cost Analysis

Cost per student

RP staff $690

AmeriCorps members $930School staff $90

Volunteer time + transport $1,520

Facilities $300

Materials/equipment $80

Total Ingredients $3,610

51

RP Cost per Student

Cost per student

Cost to school

Cost to others

RP staff $690 $320 $370

AmeriCorps members $930 $930

School staff $90 $90Volunteer time + transport $1,520 $1,520

Facilities $300 $300

Materials/equipment $80 $80

Total Ingredients $3,610 $710 $2,900

20% 80%

52

53

Cost per Cost to Cost to Cost to Cost to Cost Student School Volunteers Reading Partners AmeriCorps

IngredientsReading Partners staff 690 690AmeriCorps members 930 930School staff 90 90Volunteer time and transportation 1,520 1,520Facilities 300 300Materials and equipment 80 80Total ingredients 3,610 390 1,520 1,700 0

Fee for service ($) 320 -320

AmeriCorps grant ($) -270 270

Net cost per student (Total ingredients +fee for service + AmeriCorps grant) ($) 710 1,520 1,110 270

Portion of net cost per student (%) 20 42 31 7

Cost of Reading Partners per Program Group Student

Distribution of Cost per Student

Cost of Other Supplemental Reading Services

Total Cost Per Student

Reading Partners Other Supplemental Services

Site A $3,450 $1,840

Site B $3,420 $2,230

Site C $3,570 $2,680

Site D $5,190 $1,050

Site E $4,210 $1,980

Site F $2,740 $4,890

Pooled $3,610 $1,780

54

RP Cost versus BAU Cost

RP BAU

$710

$1,700

$2,900 $80

School funded Other resources

55

Conclusions

• Schools receive a resource rich intervention [RP] for a fraction of the cost

• Cost-effective option for under-resourced schools

• Important to consider treatment contrast and BAU in costs as well as effects.

56

SERVICE MEDIATION

CASE STUDY: CITY CONNECTS

56

Service Mediation Interventions

These SMIs have attributes of:

• Diagnosis/assessment

• Information

• Referral

• Assignment to services

• Tracking

• Service management

58

Service Mediation Interventions

Examples:• Diagnosis/assessment: Interventions to change

college remediation

• Information: Letter grades and accountability systems

• Referral: Texts to motivate class attendance

• Assignment: Grade retention

• Tracking effect: FAFSA support to direct college choice

• Tracking effect: Head Start to motivate future parental investment

• Service management: City Connects

59

Implications (1)

• Impact of intervention is the sum of diagnosis + service receipto New placement test impact depends on the test +

remedial services received based on the test result o Grade retention policy impact depends on whether

student is retained + services received by retained v. non-retained students

• Not possible to disentangle the impact attributable to the intervention from the impact attributable to service receipt; service change must be measured

60

Implications (2)

• Interventions might save resources in the long runo New placement tests have reduced enrollment in

remediation (Scott-Clayton et al., 2014)

• Interventions with no positive effect might be justified purely in resource terms

• Illustrate using economic framework 

61

City Connects (CCNX)

• CCNX diagnoses student need, refers students to most appropriate support services, manages support services, and provides some information/services to students directly

• Walsh et al. (2014b, Table 7) identify significant gains in academic achievement from CCNX

• CCNX yields average gain of 0.39 standard deviations in English Language Arts and math by 5th Grade

62

BCA of City Connects

63

Sample, Data, and Methods

• Sample: o 2 representative K-8 schools in Bostono 5 community partners per site, based on service

intensity

• Data from interviews: o Site coordinators, teachers, principals, City Connects

central office staff, and community partners

• Costs estimated with ingredients method:o Opportunity cost not accounting costs/ expenditureso Separate analyses of quantities/prices and by funder

64

 Costs for K-5 years per site(PV at K, 2013$)

  City Connects Mediated ServicesPersonnel $4,868,900Central Program Staff $174,580  School Site Coordinators $900,040  School Staff $93,140  Parental Involvement $4,990  Materials $5,040 $42,070Facilities $16,410 $186,910Other   $774,900Total $1,204,560 $5,872,800Students 780 780Average Cost $1,540 $7,530

65

Average Cost per StudentCore +

No Extra Service

Core + Full Services

Core + Estimated

Service Change

City Connects program $1,540 $1,540 $1,540

Mediated services   $7,530 $3,030

Full Cost (C) $1,540 $9,070 $4,570

Effects (E) 0.39 0.39 0.39

C/E ratio $3,950 $23,260 $11,72066

Service Mediation

• City Connects is effective – large gains in academic achievement

• If only direct costs are counted, it is very cost-effective

• But if all service costs are counted, it is unclear• Need to perform CBA to fully evaluate City

Connects• Require different impact evaluation (outcomes,

durations)

67

POWER OF TEST

CASE STUDY: EARLY LITERACY

50

Power of Test

• Before identifying impacts, specify power of test

• But economists are not focused on statistical significance

• And distribution of ratio of two variables is not same as distribution of each variable

• So use “power of economic test”

69

Early Literacy

• Estimated costs for all effective early literacy interventions

• Costs based on ingredients of effective implementations of interventions

• Related costs to effectiveness from independent studies

70

Improving Early Literacy

71

72

Cost per Effect Size Gain

K-PALS $40

Stepping Stones $570

Sound Partners $600

Reading Recovery $2,090

Fast ForWord $1,480

Corrective Reading $6,360

Wilson Reading Systems $13,390

Power of Economic Test

• How much more effective would Wilson have to be to be as cost-effective as K-PALS?

• How imprecise would estimates of these effects have to be for K-PALS to lose status as most cost-effective?

• Minimum detectable effect (MDE): the smallest true effect that has a “good chance” of being found to be statistically significant (Bloom)

• MDCE: the smallest true CE-ratio that has a “good chance” of being found to be statistically significant? 73

CA: General Conclusions

• Gives better understanding of the program• Programs typically cost more than

guesstimated or proposed or in budget documents

• Substantial variation in costs for a given program

• Most programs are not high cost given what is expected to happen

74

CEA: General Conclusions

• No expectation more costly programs are more

effective

• Very low cost programs are unlikely to be

effective

• Cost-effectiveness varies across programs,

across sites, and over time

• Targeted interventions are likely to be more

cost-effective

75

CBA: General Conclusions

• Targeted interventions are likely to have higher BC ratios

• Effective programs tend to have very high benefits

• Almost always undercount benefits• Test scores not as valuable as other skills

76

Friday afternoon! Thank you!

Brooks Bowden and Clive Belfieldwww.cbcse.org

77