PEPA is based at the IFS and CEMMAP

© Institute for Fiscal Studies

Programme Evaluation for Policy AnalysisMike Brewer, 19 October 2011

www.pepa.ac.uk

Programme Evaluation for Policy Analysis: overview

• Part of the ESRC-funded National Centre for Research Methods

• PEPA is about ways to do, and ways to get the most out of, “programme evaluation”

© Institute for Fiscal Studies

“estimating the

casual impact of”“government

policies” (although

can often

generalise)

Programme Evaluation for Policy Analysis: overview

• Part of the ESRC-funded National Centre for Research Methods

• PEPA is about ways to do, and ways to get the most out of, “programme evaluation”– Estimating the counterfactual– Characterising the uncertainty– Generalizing and synthesizing

• Beneficiaries– those who do programme evaluation– those who commission or design evaluations, or make

decisions based on the results of evaluations

© Institute for Fiscal Studies

1. Step change in conduct of programme evaluation

• Training courses, workshops, on-line resources

• Research– doing inference more accurately– social networks and policy interventions– estimating bounds of true impact

• Substantive research projects as exemplars

© Institute for Fiscal Studies

2. Maximise the value of programme evaluation

• Research– Combining behavioural models with results of evaluation– Compare RCTs with non-experimental approaches– Synthesising results

• Training courses, especially for – Those who commission evaluations and interpret results of

evaluations (link with Cross-Government Evaluation Group)

© Institute for Fiscal Studies

PEPA: who we are

• Professor Richard Blundell, UCL & IFS• Professor Mike Brewer, University of Essex & IFS

(Director)• Professor Andrew Chesher, UCL & IFS• Dr Monica Costa Dias, IFS (Deputy Director)• Dr Thomas Crossley, Cambridge & IFS• Professor Lorraine Dearden, Institute of Education & IFS• Dr Hamish Low, Cambridge & IFS• Professor Imran Rasul, UCL & IFS• Dr Barbara Sianesi, IFS• Department for Work and Pensions is a partner

www.pepa.ac.uk© Institute for Fiscal Studies

Thoughts on evidence provision

• Pilots often not designed with focus of answering “what works?”

• Funding “What works?” seen as government’s responsibility

• Data

• Limitation of “what works” evidence

© Institute for Fiscal Studies

Spare slides on PEPA

© Institute for Fiscal Studies

PEPA: overview

PEPA. Director: Mike

Brewer

1. Are RCTs worth it?

Barbara Sianesi, Jeremy Lise

2. Inference

Thomas Crossley, Mike Brewer,

Marcos Hernandez, John Ham

3. Control functions and

evidence synthesis

Richard Blundell, Adam Rosen,

Monica Costa Dias, Andrew

Chesher

4. Structural dynamic models

Hamish Low, Monica Costa

Dias, Costas Meghir

5. Social networks

Imran Rasul, Marcos

Hernandez

0. Core programme evaluation

skills

© Institute for Fiscal Studies

PEPA: research questions

PEPA

1. Are RCTs worth it?

Can non-experimental methods

replicate the results of RCTs?

How can we combine results from

RCTs with models of labour market

behaviour?

How do GE effects alter estimated

impact of training programmes?

2. Inference

Correct inference and power

calculations where data have multi-

level structure & serially-correlated

shocks?

Correct inference when policy impacts

are complex functions of estimated

parameters?

Impact of time-limited in-work

benefits on job retention?

3. Control functions and evidence

synthesis

Can we weaken control function

approach to estimate bounds?

Link between control function and

structural or behavioural model s?

How are lessons from multiple

evaluations best synthesised

4. Structural dynamic models

How best to use ex post evaluations in

ex ante analysis?

How are education decisions affected

by welfare-to-work programmes?

How do life-cycle time limits on

welfare receipt affect behaviour?

5. Social networks

How best to collect data on social

networks?

How is impact of policy affected by

the social networks within and

between treated and control groups?

Can social networks explain

heterogeneity in impact of a health

intervention?

© Institute for Fiscal Studies

PEPA: training and capacity building

PEPA

0. Core programme evaluation

skills

Core course in evaluation

methods

Courses for designers and users

of evaluations

How-to guide for PS matching

1. Are RCTs worth it?

Course on estimating “search

models”

Workshop on using “search

models”

Workshop on value of RCTs

2. Inference

Course, manual and software

tools on power calculations

Courses, survey, manual &

software for better inference

Workshop and courses on using

survivor models for policy

evaluation

3. Control functions and

evidence synthesis

Course and workshop on control

functions

Course on bounds in policy

evaluation

4. Structural dynamic models

Courses and resources for

building dynamic behavioural

models

Workshop on dynamic

behavioural models and policy

evaluation

5. Social networks

Survey and courses on using

data on social networks

Survey and courses on

collecting data on social

networks

© Institute for Fiscal Studies