Evaluation

Discovering Whether Programs Work

“Measuring” Policies & Programs

• Program Inputs

• Program Outputs

• Program Impact

Policy Inputs

• Government resources allocated to address a public “problem – Agency personnel assigned (FTEs) – Funds allocated by legislature

Policy Outputs

• Indicators of government activities in carrying out policy – School Vouchers

• # of applications for vouchers

• # of vouchers distributed

• # of students switching schools

• # of “failing schools” reinvesting in facilities, teachers, etc. – DARE

• # of children in the program • % of school districts participating

– Acid Rain • # of SO2 scrubbers installed

• # of inspections carried out

Policy Impacts

• Degree to which the policy is addressing the problem • Amount of the problem “solved”

– School Vouchers • Improved educational attainment among target population

– Using vouchers – Not using vouchers

• Reduction in the % of “failing schools” – D.A.R.E

• Reduced drug experimentation • Reduced substance use

– Acid Rain • Regeneration of aquatic ecosystems • Reduced damage to acid-sensitive structures

Policy Impacts

• Unintended/Unexpected Impacts – Vouchers

• Impoverishment of public school districts • Reduction in the educational attainment of “superio

r” schools

– D.A.R.E • Improved community-police relations

Process Evaluation

• Is the program carrying out the actions it is suppose to carry out?

• Measuring program outputs – Did the program make the public aware of the availabi

lity of vouchers? – Does the application process for vouchers encourage

participation?

– Is government distributing vouchers according to the rules?

Impact Evaluation

• Is the Program Working? – Is it reducing the problem as intended?

• Measuring Program Impact – Are test scores improving? Are fewer students droppi

ng out? • Voucher users

• Voucher non-users

– To what degree are previously “failing schools” attempting to compete – reinvesting?

Uses for Evaluations

• Go/No Go Decisions

• Improve Practices & Procedures

• Expand Program Scope and Reach

• Allocate Resources among competing programs

• Test Policy Theory

Stages of Evaluation

• Program Goal Identification

• Construction of Impact Model (Policy Theory)

• Research Design

• Data Collection

• Analysis

• Report/Presentation

Program Goal Identification

• Who wants the evaluation? – Legislature – Agency – Policy/Program directors – Program staff – Clients – Interest Groups – Scholars

Policy Theory (School Vouchers)

Research Design

• Isolating the causal effects of the program from other influences

• Measures of program impact – How would we know “change” if we saw it?

• Form of analysis – Case study – Focus groups

• Clients • Program staff

– Statistical analysis

• Data collection (sampling) requirements