Evaluation of the Intervention!\or

Realistic Evaluation after your taking action in the problem situation

PoliticianPrisons Work!

Evaluator What is it in the infinite complex

social system of prisons that make people change their behaviour

Two perspectives on reality

Fallacies of Experimental Evaluation

pre-test treatment post-test

experimental group O1 X O2

control group O1 O2

What causes something to happen has nothing to do with the number of times it happens

(Sayer, 1992, p.165)Sayer, A.R., ‘Method in social science, a realist approach’, Routledge, London, 1992

Prisons Work!

Treatment Group

Control Group

Normative Base Rate

0%

10%

20%

30%

40%

50%

60%

Recedivism Rates

Recedivism Rates

Succesionistic regularity, but what works?

Trivial?

MISQ, December 2004

ICIS, December 2007

Why Realist Evaluation• Departure: Social problems are embedded in a stratified

reality

• Rationale: Not repeating programs as a craft, but evaluating program theories as epistemic devices

• Generative causality: Knowing what works for whom in what circumstances (resources and reasoning)

• Realistic: Don’t study the program, study the inner workings

MC

xR

y

Pawson, R. and Tilley, N., ‘Realistic Evaluation’, Sage, Thousand Oaks, California, 1997

Theory-driven Evaluation

Generative Regularity

Prison Workings!Prison education (x) leads a reduction in recidivism (y)

Reasoning or Resources M1: self-realization • M2: last chance M3: second chance M4: social acceptability • M5: moral or civic responsibility M6: cognitive change

‘Certain type of prisoner’ C1: Disadvantaged • C2: The crime committed (pedophilia, pyromania) C3: First Offenders C4: Youth prisons, Isolation Cells, Federal Prisons, Death sentenced

• mediocre hypothesis

M

C

xR

y

Pawson, R. and Tilley, N., ‘Realistic Evaluation’, Sage, Thousand Oaks, California, 1997

Another Wicked Problem: Multiple Explanations

Making sense of it:• Theory adjudication

weeding out rival explanations, with focus on improvement

• Configurationalapplication in practice

• Middle-rangetransferability; patterns of repeat victimization

• Provisionaltested but not proven; soft system and wicked problems

Pawson, R. and Tilley, N., ‘Realist Evaluation’, in S Mathison (ed.), Encyclopedia of Evaluation, Sage, Thousand Oaks, California, 2004

The upside of Realistic Evaluation• Dealing with real or wicked problems• Design needs Evaluation

Did the intended outcome actually materialize?

The downside• No clear-cut recipe• Difficult language and concepts

Fitness for MIW

Camera-surveillance in Car Parks

• A form of situational crime prevention

• Visual surveillance technology monitoring variety of environments/activities

• Growth estimated at 15-20% per year (Davies, 1996)

• In car parks CCTV is used to prevent and reduce crime

Opdracht

• Experimentele Evaluatie

Camera’s werken!

• Realistische Evaluatie (opdracht):

Wat is het in cameratoezicht waardoor mensen hun gedrag aanpassen – Bedenk de mechanismes– Bedenk de contexten

Realistische Analyse

Mechanismen• Caught in the act• You’ve been framed• Nosy parker• Effective deployment• Publicity mechanism• Time for crime• Memory jogging• Appeal to be cautious

Context• Criminal clustering• Style of usage• Lie of the land• Alternative targets• Resources• Surveillance Culture

Do you believe that

Hypothesizing about contexts, mechanisms and outcomes and testing these empirically

Is more valuable than

Comparing the rates of crime before and after CCTV

Then we succeeded