UK ERA UK Employment Retention and Advancement Demonstration

UK ERAUK Employment Retention and Advancement Demonstration

Final Impact and Cost-Benefit Results

Policies Studies Institute SeminarLondon, England6 December 2011

2

Test a new model to help low-income groups keep jobs and advance in labor market

• Little attention was paid to these issues in British or US welfare-to-work policy

• New demo inspired by US example (ERA)

• “Hunches,” but no evidence on what did/didn’t work

Help build capacity in UK to conduct large-scale random assignment experiments

Not your typical British “pilot”

Goals of the UK ERA demonstration2003-2011

3

Evaluation sponsor• UK Department for Work and Pensions (DWP)

US Research Partner• MDRC (NYC-based, non-profit social policy research

organization; leader in RCTs)

UK research partners• Policy Studies Institute (PSI)• Institute for Fiscal Studies (IFS)• National Institute for Economic & Social Research (NIESR)• Office for National Statistics (ONS)

Lessons traveled in both directions!

Transatlantic collaboration

4

Unemployed and entering New Deal welfare-to-work program:

(1) New Deal Lone Parents (NDLP)

(2) New Deal 25-Plus (ND25+)

Working part-time (lone parents):(3) Working Tax Credit (WTC) group

UK ERA Target groups

5

Operates within Jobcentre Plus centers• Separate staff• In some offices, separate post-employment teams

33-month program: • If unemployed at intake, get job placement

through New Deal w-t-w program (~ 9 months)

• Post-employment for all who work: 24+ months of “in-work” job coaching and support from “Advancement Support Advisors”

24 months of financial incentives

UK ERA model

6

Retention bonus £ 400 3 times/yr for for sustained full-time work 2 yrs (£ 2,400 total)

Tuition assistanceif combine training + work £ 1,000

Tuition bonus if complete training up to £ 1,000

In-work emergency funds £ 300/worker

Financial incentives

7

6 districts (Total N = 16,000 people):• 4 districts in England

– (E. Midlands, London, NE England, NW England)• 1 in Scotland• 1 in Wales

Spread across 58 local offices (where random assignment took place)

The pilot sites

8

Implementation, impact, benefit-cost analysesAdministrative records: up to 5 years of follow-up

• Employment and earnings (Labor Market Information Survey)• Benefit receipt• Bonus payments

3 waves of client surveys • 1-year for all • 2-year and 5-year for lone parents

Qualitative observations and interviews• ERA staff and participants

Cost data

Comprehensive evaluation

NDLP lone parents • Many who entered work wished to defer full-time work or

work+ training because of caring responsibilities, especially if had younger children

WTC lone parents• Many attracted to ERA because of offer of training support• Not all wanted full-time work

ND25+ • When they worked, tended to work full time• Many didn’t want deep ongoing engagement with ERA or JP+

What did participants want?

Target group #1

New Deal for Lone Parents (NDLP)

10

11

– Not working at baseline

– Receiving Income Support

– Volunteered for New Deal welfare-to-work program (and ERA)

New Deal for Lone Parents (NDLP)

12

NDLP: Lone parents not working at baseline

Random assignment

New Deal intake

ERA

Control

ERA New Deal

Welfare-to-work +advancement focus

Regular New Deal

Welfare-to-work

In-work support + incentives

JOB

JOB

Random assignment

PRE-employment

POST-employment

Implementation: Big challenge for JC+

New Deal phase: ERA vs. Control “treatment difference” minimal beyond offer of incentive/support

Post-employment: Struggles providing meaningful “in-work” services, but improvement over time

• Primacy of placement goals of JC+• Steep “learning curve” for advisers• Challenges maintaining contact with working participants

Still, a “fair test” post-employment model • More “in-work” help/advice for ERA group than controls• Most ERA participants aware of incentives offer

14


Got “help/advice” from JC+ while employed (years 1 – 2)

ERA Control

61%

19%

Base = WORKERS in each group

Types of advancement issues

• Getting better job

• Increasing hours

• Getting a promotion

• Setting career goals

• Help finding ed/training

Among workers: Got help/advice on ANY advancement issue while working?

15


Impacts on training/education (years 1-2)

Obtained any training

qualifications

55.7

23.0

ERA Control ERA Control

Participated in training or education

16



Obtained any training

qualifications

60.6

24.6

55.7

23.0

+ 4.8 pp**(+9%)

+ 1.7 pp (ns)


Participated in training or education

17


Impacts on employment

Ever worked

in 5 years

Months workedin 5 years

Ever worked full timein years 1-2

79.0%

25.0 27.9

ERA Control ERA Control ERA Control

18



Ever worked

in 5 years


Ever worked full timein years 1-2

79.0%

25.137.5

79.0%

25.0 27.9

+ 9.6 pp***(+ 34%)



Impacts on earnings trends

2005-2006 2006-2007 2007-2008 2008-20093000

3500

4000

4500

5000

5500

6000

19

Earn

ings

(£)

Tax Year

Control group



2005-2006 2006-2007 2007-2008 2008-20093000

3500

4000

4500

5000

5500

6000 ERA programme ended for all by October 2007 (earlier for some)

20

Earn

ings

(£)

Tax Year

Control group

ERA group

21


Impacts on earnings, by tax year4-year total

£3,862 £4,999

£17,280

£3,554 £5,033

£16,742

- £ 35 (ns)

+ £ 538 (ns)


+ £ 308 ** (+ 9%)

Early (2005-6) Last (2008-9)

22


Impacts on 4-year total earnings, by education subgroup

Lowest (No qualifications)

Highest (A-level)

Middle (GCSE)

£12,923£16,183

£22,845


23


Impacts on 4-year total earnings, by education subgroup

Lowest (No qualifications)

Highest (A-level)

Middle (GCSE)

£12,289£15,546

£26,383

£12,923£16,183

£22,845


- £ 634 (ns)- £ 637 (ns)

+ £ 3,537** (+ 16%)†††

24

Not because of increase in training!• Impact on training = 0.2 %• Impact on earnings = 15.5 %

Hypothesis: ERA tapped their “reserve capacity”• At entry: Not working, but interested in work – although

some reluctance to work full time

• With A-level qualifications, if they chose to work, they’d have better job opportunities

• ERA incentives and support may have gotten some to try full-time work


Why did A-level subgroup do better?

25

Examined through cross-office analysis (multi-level model)• 58 offices in which random assignment occurred; 37 with

sufficient sample/data for comparison

• Did variation in implementation influence size of ERA’s office-level impacts on work and welfare – after holding constant x-office differences in parents’ characteristics?

Found larger impacts in offices where: • Greater participant awareness of incentives (better marketing)• More in-work help received (vs. controls)• Thus: Both services and incentives mattered – but incentives

may have mattered somewhat more


Incentives vs. services

26

Ethnic minority findings: Bigger effects?• Exploratory subgroup analysis – small samples • NDLP (and WTC): Larger impacts on work and earnings

and reductions in benefits for ethnic minorities than for whites – not always statistically sign., but a pattern

• ERA countered lower-expectations for this group?

Few non-economic impacts on parents or children • Little evidence of positive – or negative – effects on

overall well-being


Other findings

Target group #2

Working Tax Credit (WTC)Lone Parents

27

28

• Already working part-time (16-19 hours/week)

• Receiving Working Tax Credit

• More attached to labour market and more advantaged in general

• Volunteered for ERA, beginning with post-employment phase

Working Tax Credit (WTC) lone parents

29

WTC: Lone parents working part-time at baseline

Random assignment

ERA

Control


JOB

Random assignment

POST-employment

Applyfor ERA

30


Got “help/advice” from JC+ while employed (years 1 – 2)

ERA Con-trol

75%

7%







• Help finding ed/training


31



Participated in training

or education

72.2

32.6

59.6

28.0

+ 12.6 pp***(+21%)

+ 4.5 pp**(+16%)


Obtained any training qualifications

32



Ever worked

in 5 years


Ever worked full-time in years 1-2

88.1%

38.5 41.5%

86.8%

38.330.0%

+ 1.3 pp (ns)

+ 0.2 months (ns)

+ 11.5 pp***(+ 38%)




2005-2006 2006-2007 2007-2008 2008-20096,000

6,500

7,000

7,500

8,000

8,500

9,000

33

Earn

ings

(£)

Tax Year

Control group



2005-2006 2006-2007 2007-2008 2008-20096,000

6,500

7,000

7,500

8,000

8,500

9,000

34

Earn

ings

(£)

Tax Year

ERA group

Control group

ERA programme ended for all by October 2007 (earlier for some)

35


Impacts on earnings, by tax year

Early (2005-6) Last (2008-9) 4-Year Total

£7,069 £8,636

£30,615

£6,667 £8,401

£29,695+ £ 402 ** (+ 6%) + £ 234 (ns)

+ £ 921 (ns)


QualificationsSubgroup

ERA group

Control group

Difference (Impact) % Change

Lowest (No qualifications) 55.3 42.3 13.0 ** 30.7%

Middle(GCSE) 72.9 55.6 17.4 *** 31.3%

Highest (A-Levels) 79.8 71.6 8.2 ** 11.5%

Full sample 72.2 59.6 12.6 *** 21.1%

36


Impacts on training, by subgroup(Took training or education course, years 1-2)

37

-20

0

20

40

60

80

100

**

***

Impact on “took any courses” (21.1% ***)

*** **

% C

hang

e Im

pact

Impact on “obtained qualifications” (16.2% **)

*****


Payoff to impact on training?

38

-20

0

20

40

60

80

100

**

***

Impact on “took any courses” (21.1% ***)

*** **

% C

hang

e Im

pact

Impact on “obtained qualifications” (16.2% **)

NO Impact on total 4-year earnings (3.1% ns)

*****

WTC: Lone parents working part-time at baselineNo association between

training impacts and earnings impacts

Were courses occupationally relevant?

Some for personal enrichment or hobbies • E.g. aromatherapy, flower arranging

Some “safety” or non-advancement on the job• (e.g., CPR)

Course content analysis:• Verbatim responses to the 12-month and 24-

month surveys were analyzed and coded

• Developed training typology39

40

Type of courseERA

groupControl

group ImpactTrade-Specific (%) 68.0 54.5 13.5 ***

Business and information technology 14.6 10.7 4.0 **

Computer user skills 23.9 19.2 4.7 **

Personal services/retail 13.3 10.9 2.5

Skilled trades/protective services 3.0 1.0 2.0 **

Social services, child care. Education 28.6 26.7 1.9

.WTC: Lone parents working part-time at baseline

Impacts by type of trade-specific course

41

Type of courseERA

groupControl

group Impact

Not trade-specific (%) 34.0 28.1 5.9 **

Basic skills 7.9 4.7 3.2

Soft skills 6.5 5.5 1.1

Workplace skills 21.2 18.0 3.2

.WTC: Lone parents working part time at baseline

Impacts by type of “general” course

42

Possible reasons:

• Unrealistic expectations about “dream job”? Not driven by demand for workers

• Advisers were employment ‘generalists’

• No assistance in making transition between specific training and specific job openings


So why no payoff to training impacts?

Converting training into advancement

43

Increase training

Relevant courses

Connection to

better jobs?

Target group #3

New Deal 25 Plus (ND25+)

44

45

• Claiming Jobseeker’s Allowance (JSA) for at least 18 months

• Participation in ND25+ is compulsory

• Participation in ERA experiment voluntary

New Deal 25 Plus (ND25+)

46

ND25+: Long-term unemployed at baseline

Random assignment

New Deal intake

ERA

Control

ERA New Deal

Welfare-to-work +advancement focus

Regular New Deal

Welfare-to-work


JOB

JOB

Random assignment

PRE-employment

POST-employment

47


Got “help/advice” with advancement from JC+ while employed (years 1 – 2)

ERA Con-trol

47%

12%







• Help finding ed/training(NDLP ERA = 61% )(WTC ERA = 75%)


0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 600

10

20

30

40

50

60

70

80

90

100

ND25+

NDLP22%

43%

Months after Random Assignment

% E

mpl

oyed


Control group employment rates

WTC61%

49



Ever workedIn 5 years


Series1

55.1%

13.2


50



Ever workedIn 5 years

+ 1.1 months** (+ 8%)

+ 2.2 pp* (+ 4%)


Series1

57.3%

14.3

55.1%

13.2


2005-2006 2006-2007 2007-2008 2008-20092,000

2,500

3,000

3,500

4,000

4,500

5,000

Tax Year


Impacts on earnings trendsEa

rnin

gs (£

)

Control group

2005-2006 2006-2007 2007-2008 2008-20092,000

2,500

3,000

3,500

4,000

4,500

5,000

Tax Year

Control group


Impacts on earnings trendsEa

rnin

gs (£

) ERA group

ERA programme ended for all by October 2007 (earlier for some)

53


Impacts on earnings, by tax year

Early (2005-6) Last (2008-9) 4-year total

£3,070 £3,954

£14,162

£2,758 £3,591

£12,681£ 312 **(+ 11%)

£363**(+10%)


£1,481pp ** (+ 12%)

54

Amount received

in 5 years


Impacts on Jobseeker’s Allowance


24.9 26.4

- 1.5*** months (-6%)-2.4** pp (-8%)

Received any in month 60

£7,493

£ 7,067

Months receivedin 5 years

28.3% 30.7%

- £ 426** (- 6%)

55

23% of eligible ND25+ were not included in study• 9% “not offered” opportunity to participate• 14% were offered, but refused

Do impact estimates represent likely effects on full eligible population?

• Explored with non-experimental techniques

Finding: Impact estimates may be conservative• Had experiment included full eligible population, impacts on

employment and earnings for ND25+ group may have been larger


Assessing generalisability

Cost-benefit findings

56

• Cost per participant for ERA and related services

• Net government investment in ERA per participant, above and beyond cost of normal control group services

• Accounting perspectives – Did the ERA participants come out ahead?– Did the Exchequer/taxpayers come out ahead?– Did “society” (the combination of participants and

Exchequer) come out ahead?

• Net present value– Sum of all economic “gains” and “losses” per perspective– Estimated in 2005-2006 pounds; 3.5% discount rate

57

Goals of the cost-benefit analysis

Base estimate: 10-year time horizon • 5 years observed• 5 years projected

- Applied annual impact decay rates – varied by target group and outcome measure

Alternative A: 5-year time horizon • 5 years observed• No projection

Alternative B: 20-year time horizon • 5 years observed• 15 years projected 58

Time horizons for assessing costs and benefits

• ERA budget cost and cost of external training

59

Gross costs

ND25+ NDLP WTCERA costs £1,860 £1,979 £2,640

• Pre-employment 26% 28% 0%• Post-employment 74% 72% 100%

Total 100 % 100 % 100 %

Incentives 35% 37%

44%

• Subtract costs of control services from gross costs• Compare these costs with “benefits”

60

Net costs

ND25+ NDLP WTCNet costs for ERA £1,261 £1,236 £2,625

• Pre-employment 0% 0% 0%• Post-employment 100% 100% 100%

61

Illustration: Distribution of gains and losses across perspectives

Participants Exchequer SocietyGross earnings + 0 +Income taxes - + 0Reduction in benefit payments

- + 0

Program costs 0 - -Etc.

Etc.

Net gain or loss (NPV) ? ? ?

62

Target Group Participants Exchequer

ND25+ + 726 + 1,816(ROI = 4.01 to 1)

NDLP - 37 -124

• NDLP A-level subgroup + 2,074 + 1,479

(ROI = 3.92 to 1)

WTC + 230 - 1,836

Cost-benefit results(Net Present Value per ERA participant, in £ )

Alternative assumptions applied:• Discount rates: 2%, 5%• Annual earnings impact decay rates: e.g.,

0%, 20%, 34% decay, depending on group• Time horizon: 5 years, 20 years

Findings:• Results not highly sensitive to alternative

assumptions• Basic story remains the same

63

Sensitivity tests

Considerable effort to assess omitted benefits and cost such as:

• Substitution effects• Leisure bias• Effect on health status• Effects on quality of life

Analysis suggests that, in general, these effects appear likely to be small in magnitude

64

Consideration given to“hard-to-monetarize” effects

ND25+ results are impressive • Unusual to see a “win-win” – i.e., net gain from

participant and government perspective

NDLP results:• Break-even for full group• “Win-win” for A-level subgroup, highlighting

potential for ERA to be a “good investment” for some lone parents

WTC results:• Sizable loss for the government for marginal

benefit for participants; not a good investment65

Lessons from cost-benefit

Final Reflections

66

67

On policy:• UK results (like US experience) show difficulty of

achieving advancement goals

• “Types of participants” matter a lot: UK ERA strategies can be good investment for some

• Overall: Much evidence to build on – new policies in this area won’t start from scratch

• Much value in learning what didn’t work

• Good that ERA wasn’t rolled out as national policy for all 3 target groups before the evidence was in!

Final reflections

68

On evaluation:• Can see the value of reliable control groups – and that

‘outcomes’ alone can give wrong answer

• Can see value of long-term follow-up: stopping too soon would have given wrong answer

• ERA and the transatlantic exchange helped build capacity on random assignment among civil service staff as well as research partners

• Dialogue around “evidence-building” now is very different than it was when ERA was conceived

Final reflections (continued)

Lessons from the US Employment Retention and Advancement Project

70

What is US ERA?• National study of programs aiming to

promote stable employment and wage progression– mostly current and former welfare

recipients/single parents

• Tested 12 distinct models in cities in 6 states– Little known about how to improve employment

retention and advancement outcomes

• Evaluation at final stage: most programs have 3-4 years of follow-up

71

Summary of US ERA impacts

• 3 models have shown positive economic impacts

• 9 models have shown no or only sporadic impacts

Features of the 3 promising modelsModel Pre-

employment services

Post-employment services

Post-employment financial incentives

Corpus Christi (Texas)

Chicago(Illinois)

RiversidePASS(California)

72

73

Corpus Christi (Texas) ERA programme—Most Like UK ERA for NDLP group—

• The model: Job search, financial incentives for retaining full-time jobs

• Targeted lone parents: TANF applicants/recipients (like Income Support)

• Providers: Non-profit organizations

• Control group: in relatively strong welfare-to-work program (mandatory)

Impacts on earnings Corpus Christi, Texas

74

RA 1

**2 *3 4 5 6 7

*

8 9

**10

***

11

***

12

* 1

3

**14

**15

**16

0

500

1,000

1,500

2,000

2,500

3,000

Quarter after random assignment

($)

ERA group

Impact on 4-Year total earnings = +2,560 (15%)***

Control group

Chicago and Riverside (California) ERA Programmes

• Neither used financial incentives

• Both put strong emphasis on re-employment assistance

• Both relied on contracted service providers– Chicago: for-profit provider with links to

employers in health and security sectors– Riverside: 3 non-profits, 1 community college, and

1 local welfare office 75

Impacts on earnings Chicago

RA 1 2 3 4 5 6

**

7

**

8

* 9 10 11

* 1

2

** 1

3 14

* 1

5 16

0

500

1,000

1,500

2,000

2,500

3,000


($)

Control group

Impact on 4-Year total earnings = + $1,909 (7%)*

ERA group

Impacts on earnings Riverside PASS

77

** R

A 1

***

2

***

3

***

4

**

5

**

6

* 7

**

8

* 9 10

* 11

* 12 13

* 1

4

** 1

5

***

16

0

500

1,000

1,500

2,000

2,500

3,000


($)

Impact on 4-Year total earnings = +3,470 (10%)***

ERA group

Control group

Evidence on other strategies: Not promising

• Lowering caseloads to increase individualized career guidance

• Holding out for job in preferred field versus first job

• Service continuity, job coaching from pre-to post placement w/o other services

• Reducing work hours requirements to encourage employed welfare recipients to participate in education and training

78

Not promising (continued)

• Partnerships between welfare and workforce agencies to leverage resources and experience

• Providing retention services at the worksite (mostly nursing homes)

• Locate and provide retention & advancement services to former benefits recipients

79

Common theme among ineffective programs

• Counseling and referrals alone do not appear to be effective

• Need post employment services PLUS concrete, “tangible” services:– Financial incentives OR– Reemployment services w/strong job

development/employer ties

80

WorkAdvance

A Next Generation Advancement Strategy

WorkAdvance Partners

A a collaboration between:• MDRC• NYC Center for Economic Opportunity (CEO)

(An anti-poverty innovation unit in office of Mayor Bloomberg)• Mayor’s Fund to Advance NYC

Being piloted in 4 cities in 3 states• NYC , Cleveland and Youngstown (Ohio), and Tulsa (Oklahoma)

Funded with federal and private philanthropic dollars as part of Obama’s ‘Social Innovation Fund’

82

WorkAdvance model: A hybrid vision

83

Sector-basedstrategies

Post-employmentStrategies

(ERA-inspired)

+

Common features of sector strategies

• Training programmes targeted to specific careers in specific industries

• Heavy screening re: participants’ ‘fit’ for sector

• Employer input on programme’s approaches

• ‘Pipeline’: Staff have close ties with employers, helping to link training and the job

• But limited post-employment services84

Evidence on sector approaches

Study by Public/Private Ventures (P/PV) shows

• Tested sector training programs in 3 cities

• Eg, manufacturing, medical billing, computer recycling

• Small random assignment study showed substantial positive impacts on employment and earnings 18% gain over 2 years; 29% in 2nd year (after training)

85

’Tuning in to Local Labor Markets: Lessons From the Sectoral Employment Study.’ PPV. Maguire et al. 2010http://www.ppv.org/ppv/publications/assets/325_publication.pdf

P/PV study: Impacts of sectoral employment programme

http://www.ppv.org/ppv/publications/assets/325_publication.pdf

’Tuning in to Local Labor Markets: Lessons From the Sectoral Employment Study.’ PPV. Maguire et al. 2010http://www.ppv.org/ppv/publications/assets/325_publication.pdf

P/PV study (continued)

http://www.ppv.org/ppv/publications/assets/325_publication.pdf

Open questions about sector approach

• Can P/PV results be replicated? (Small study)

• Can other sectors achieve same results?

• Will the results hold up in the long term?

• Do participants remain in the sectors they trained for? Remain on a career tracks?

• Cost-benefit?

• Would adding a substantial post-employment component strengthen the effects? 88

WorkAdvance Demonstration

Sector-based training & job placement

• Narrow range of sectors and occupations

• Staff have deep expertise and employer connections

Strategic post-employment follow-up

New random assignment test• Implementation, impact, and cost-benefit

evaluation89

5 industry sectors

• Information Technology

• Manufacturing

• Transportation

• Environmental Remediation

• Health

90

Study eligibility criteria

• 18 years or older• Working < $15/hr in current or previous job• Family income <200 % of federal poverty level• Committed to a career in targeted sector• Meet sector-specific requirements (literacy tests, drug/criminal screening, manual dexterity etc.)Limitation: Many in US ERA would not qualify

91

Pre-employment services and training

Pre-employment services: • Sector-focused job readiness training; work with

advancement coach to start individual career plan; assess other needs

Occupation-specific skills training• Focused on particular industry career tracks, skill

needs, and training curricula identified by employers

92

Job development

Job development specialists:

• Identify job openings—who’s hiring within the sector

• Understand employers’ skill needs in targeted industry

• Have relationships with specific employers

93

Post-placement assistance: Building on ERA

Like ERA – but with a sector “twist”:• Help participants identify career advancement

opportunities

• Advise on further relevant training, as appropriate

• Assist with job changes for better opportunities or fit

• Rapid re-employment for those losing jobs

• Employer feedback on participants’ performance

Some participants (in some sites) may only get:• Sector placement + post-employment services 94

Next steps

• Random assignment getting underway

• Refinements to the model still underway, especially post-employment component

• Lessons from UK ERA are informing those approaches and evaluation strategy

• Federal government (HHS) is launching other RCT tests of health sector training programmes – opportunity for future comparisons

95

Documents

UK ERA UK Employment Retention and Advancement Demonstration