Designing a Random Assignment Social Experiment In the U.K.; The Employment Retention and

Advancement Demonstration (ERA)

My Role in ERA

• A member of the 6-person team in the British Cabinet Office that:– Designed the ERA program– Designed the evaluation of ERA to determine

how effective it is

• A member of the team evaluating ERA– I’m in charge of the cost-benefit analysis

My Talk Today

• How it was designed

• What the ERA program is

• The ERA evaluation and how experimental methods are being used to evaluated it

• Some early results

• Some lessons from designing ERA (if time permits)

Employment Retention and Advancement Demonstration (ERA)

• Design Work in 2001-2002• Run as pilot program in 2003-2007 in 6 sites• Analysis conducted: 2007-2010

Why Was ERA Undertaken?• To test a program that tries to keep low-wage

workers employed after they find jobs and help them advance

• To promote the use of random assignment experiments in the U.K.

• Because it was expected the value of the information obtained about the program would exceed the cost of the obtaining the information (“the rational paradigm)– This requires that the information is actually used

Program Features

• Continued contact with Jobcentre Plus advisors (ASAs) after obtaining employment

• Retention bonus of £400 every 17 weeks if works full-time for 13 weeks

• Training bonus of £8 per hour of training

• Must contact ASA to receive bonus

• ASAs advise participants on job advancement

Unique Features of ERA

• Design of Program and Evaluation done simultaneously

• Design work done at British Cabinet Office• Expectation that decision on national

implementation of pilot would be based on evaluation findings

• Pilot run as large-scale random assignment experiment

• Focus is on what happens after a job is obtained

Evaluation Components

• Process or implementation study

• Impact analysis

• Cost study

• Cost-benefit analysis

Project Characteristics

• Long planning period – Absence of political pressure

• Developing the program and evaluation designs in tandem– Allowed for a good evaluation design

• Designing the project in the U.K. Cabinet Office– Permitted project team to focus on the project– Need to transfer project to DWP once designed

• Random assignment

How Does Random Assignment Work?

• Estimate the impact of a policy, a change in a program, or a intervention

• Provide evidence of whether the policy has led to (or caused) the change it was designed to - a causal link!

• The overall objective - to provide policymakers with evidence of whether their policy works

Measuring Impact

• Experiments measure the impact of policies/interventions in terms of their impact on outcomes;

• E.g. Does ERA increase earnings?

• The outcome measure is the earnings of the program group

Establishing Causality

• To establish that a policy or intervention has caused change to occur rather than some other factor – E.g. many factors will affect individuals earnings in addition

to participation in ERA

• If we find that earnings have increased among the program group, how do we rule-out the influence of other factors?– E.g. most persons who entered ERA were not employed, but

some would inevitably have found jobs without ERA.

• We do this by estimating what we call the ‘counterfactual’—what would have happened without the program

The Counterfactual• Is what would have occurred in the absence of

the policy or intervention– E.g. what would have happened to earnings over the

same period of time for the same individuals had ERA not existed?

• This is unobservable or missing information

• We have to estimate the counterfactual - that is determine what would have happened in the absence of ERA

Estimating the Counterfactual

• Wide range of ways to do this• These vary in complexity, rigor and the degree of

control required by those planning the evaluation• We will look at only one method: The simple two

group random assignment experiment• When feasible, random assignment is the best

method for estimating a counterfactual

Random Assignment

• In theory the most rigorous way to assess the impact of a policy or intervention

• Provides unbiased estimates of intervention impacts• How does it work?

– You identify individuals or groups who are eligible for a new intervention or policy

– You create two groups at random - intervention and control groups (essentially a computer flips a coin)

– Intervention group receives the new service or intervention, control group does not.

Simple Randomised Experiment

Intervention

Eligible

population R

Intervention group

Control

group

Outcome

= O1

Outcome

= O2

• the counterfactual is simply ‘O2’

• policy impact is ‘O1’- ‘O2’

Random Assignment

• In ERA, half of those willing to be randomly assigned were randomly assigned to the program group and half to a control group

• Those assigned to the control group could not receive the services and financial incentives provided

• The baseline information collected for ERA at the point of random assignment indicates that the two groups are very similar in terms of all observable measures– randomisation worked

Random Assignment Continued

• After entering ERA, employment, earnings, and other outcomes of the program group are compared for several years to those of the control group during the same years

• Randomised evaluations of social programs used frequently in US (over 200 times)

• It is increasingly used in other countries, especially developing countries; but so-far it has been used far less often than in the U.S.

An Illustration

• At random assignment about 25% of one of the ERA program groups worked

• 24 months after random assignment, about 55% of the program group worked

• Was this increase due to ERA?

Illustration (continued)

• 24 months after random assignment, about 52% of the control group worked

• Thus, only about a 3 percentage point increase in employment (55% - 52%) is attributable to ERA

Advantages of Random Assignment

• As intervention and control groups were created randomly, they are statistically equivalent

• Equivalent in both what we can observed about them and what we cannot

• At follow-up, when me measure our outcome variables, the only difference between the two groups is the impact of the intervention

• In theory provides unambiguous results• No need for complex statistics as with other

methods - ease of interpretation• Baseline information not essential, but helpful

Disadvantages of Random Assignment

• On its own, only provides a measure of average impact - policy makers may have other questions about the policy

• Can be expensive and complicated to implement• Sometimes impractical to implement• Possible lack of generalizability • Can create political problems by denying services to

controls (but if resources are limited some method must be used to deny some)

• In many cases, can take time for results to emerge• Other evaluation designs may also be subject to these

limitations-- depends on what is being evaluated

Conclusion

• Trade off -- rigour against difficulty in implementation

• Random assignment can be expensive - policy budget!

• Experimental methods require lots of data• Only answers certain types of questions• Still, when it is feasible, random assignment

provides the most accurate estimates of impacts

Some Early Findings(* = statistically significant impact)

NDDP WTC

Program Group

Control Group

Impact Program Group

Control Group

Impact

Ever worked

Year 1 65.3% 59.7% 5.7* 97.6% 95.9% 1.7*

Year 2 67.5% 62.9 4.6* 95.8% 94.6% 1.2

Months Worked

Full time 5.2 3.5 1.7* 7.3 5.1 2.2*

Part time 7.0 7.5 -0.5 14.5 16.5 -2.0*

Earnings (£)

Year 1 3,612 2,764 849* 8,255 7,916 339

Year 2 4,781 4,108 673* 8,962 8,458 503*

Lessons from the ERA Design Work

• Developing ownership among those fielding and running a program important

• Randomized experiments are feasible, but circumstances must warrant their use

• Use of multi-disciplinary teams should be encouraged

• Designing programs and evaluations in tandem should be done whenever possible