S08 - Sanderson, 2003

8/10/2019 S08 - Sanderson, 2003

1/16

This article was downloaded by: [Universit du Qubec Montral]On: 06 September 2014, At: 01:55Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Research Papers in EducationPublication details, including instructions for authors andsubscription information:

http://www.tandfonline.com/loi/rred20

Is it what works that matters?

Evaluation and evidencebased

policymakingIan Sanderson

Published online: 24 Jun 2010.

To cite this article:Ian Sanderson (2003) Is it what works that matters? Evaluation

and evidencebased policymaking, Research Papers in Education, 18:4, 331-345, DOI:

10.1080/0267152032000176846

To link to this article: http://dx.doi.org/10.1080/0267152032000176846

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (theContent) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content

should not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoever orhowsoever caused arising directly or indirectly in connection with, in relation to or arisingout of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
http://www.tandfonline.com/page/terms-and-conditionshttp://dx.doi.org/10.1080/0267152032000176846http://www.tandfonline.com/loi/rred20http://www.tandfonline.com/page/terms-and-conditionshttp://www.tandfonline.com/page/terms-and-conditionshttp://dx.doi.org/10.1080/0267152032000176846http://www.tandfonline.com/action/showCitFormats?doi=10.1080/0267152032000176846http://www.tandfonline.com/loi/rred20

8/10/2019 S08 - Sanderson, 2003

2/16

Research Papers in Education ISSN 0267-1522 print/ISSN 1470-1146 online 2003 Taylor & Francis Ltdhttp://www.tandf.co.uk/journals

DOI: 10.1080/0267152032000176846

Research Papers in Education 18(4) December 2003, pp. 331345

Ian Sanderson is Director and Professor of Policy Analysis and Evaluation, Policy Research

Institute, Leeds Metropolitan University.

Is it what works that matters?Evaluation and evidence-based

policy-making

Ian Sanderson

ABSTRACT

The notion of evidence-based policy-making (EBP) has gained renewed currency in the UK

in the context of the current Labour Governments commitment to modernise government.

Thus, a key driver of modernisation is seen as evidence based policy-making and service

deliverywhat matters is what worksin the context of a performance management

strategy for regulation of public services. The aim of this paper is to critically examine the

assumptions underpinning EBP asking, in particular, the extent to which the increased

emphasis on the role of evidence in policy-making is indicative of instrumental rationality

which erodes the normative basis of policy-making and undermines the capacity for

appropriate practice. The potential for theory-based evaluation to deliver on its evidential

promise is critically examined and, based upon an expanded notion of practical reason, it is

argued that we need to extend the scope of our concern from what works to what is

appropriate in addressing complex and ambiguous social problems, embracing ethical-moral

concerns.

Keywords: evidence; policy; evaluation; theory-based; rationality

. . . (I)n the degree in which an active conception of knowledge prevails . . . (c)hange

becomes significant of new possibilities and ends to be attained; it becomes prophetic ofa better future. Change is associated with progress rather than with lapse and fall. Since

changes are going on anyway, the great thing is to learn enough about them so that we be

able to lay hold of them and turn them in the direction of our desires. (Dewey, 1957,

p. 116.)

8/10/2019 S08 - Sanderson, 2003

3/16

332 Research Papers in Education Volume 18 Number 4

INTRODUCTION

The role of scientific knowledge in public policy-making is an issue of enduring concern to

both policy makers and researchers, a legacy of the Enlightenment in the seventeenth century

when the focus shifted to the potential for the application of knowledge to change and

improve the world. A key figure in what A. N. Whitehead called the century of geniuswas

Francis Bacon who, Zagorin (1998, p. 222) argues, . . . gave science an ethos and socialfunction, the investigation of nature for human betterment which, if never universally

accepted, continues to be very widely regarded up to the present day as its ultimate rationale.

John Dewey, in his quest for a philosophical basis for the capacity to turn change in the

direction of our desires, was a passionate advocate of Bacons systematic critical empiricism

founded upon careful experimentation to . . . force the apparent facts of nature . . . (to) . . .

tell the truth about themselves, as torture may compel an unwilling witness to reveal what he

(sic.) has been concealing(Dewey, 1957, p. 32).

In contrast to Deweys optimism about the role of reason in attaining the better and

averting the worse(Dewey, 1993), the pessimistic diagnosis of some postmodernistsabout

a world out of control and beyond the reach of redemptive politics (OSullivan, 1993)appears rather bleak. Of course, there are two issues here. The first concerns our capacity to

understand the worldto wrest those facts from the uncooperative witness; the second

concerns our ability to apply those factsto influence the direction of social change in the

context of wider political drivers. Enough has been written about policy-making processes to

show that ideas rarely win the battles, although there remains a deep-seated optimism about

their capacity to help win the wars.

The position adopted in this paper is one of constructive scepticism in a critical appraisal

of the present Governments pre-occupation with evidence-based policy-making (EBP). Tony

Blair set the agenda shortly after being elected for his first term of office by outlining the

commitment to modernisethe public sector and declaring that what counts is what works.

In other words, the old age of ideologically-driven politics was to be consigned to the dustbin

of history and a new age of modern policy-making would be driven by research evidence of

what was proven to be effective in addressing social problems and achieving the desired

outcomes. In key policy areas such as crime, education and welfare-to-work, we continually

hear the Prime Minister and other ministers talk about their commitment to finding out what

works.

Now, for an applied social researcher, this position raises a dilemma. It is music to the ears

of the applied researcher committed to the modernist belief in social progress informed by

reason and immersed in contracted research from government departments and agencies

specifically intended to contribute to the development and improvement of public policies.

However, the music is mixed with distant alarm bells in the ears of the academic socialscientist influenced by the work of, for example, Jurgen Habermas, John Dryzek and Frank

Fischer (Sanderson, 1998, 1999). Is this emphasis on knowledge and expertise in policy-

making a form of instrumental rationality, focusing on deriving correct means to given ends

at the expense of consideration of the appropriateness of those ends? Does it signal the

devaluing of democratic debate about the ethical and moral issues raised by policy choices?

How much does what worksmatter? How much emphasis can and, indeed, should be

placed on scientific evidence of what works in decisions about policies to address social

problems? This paper seeks to address these questions by the following route. First, I will refer

briefly to the historical context of EBP and outline in a little more detail the present

8/10/2019 S08 - Sanderson, 2003

4/16

Is it what works that matters? 333

Governments stance. I then focus on the role of evaluation in providing the evidence

required, asking to what extent it can tell us what works. Based upon the conclusion that the

rationalist conception of EBP has some rather shaky foundations, I question just how much

what worksreally does matter in the context of a broader conception of policy-making. In

the concluding part, I elaborate briefly a proposition on what does matter in making public

policy choices.

WHAT WORKS: THE NEW HOLY GRAIL OF POLICY-MAKING?

Faith in the power of reason to guide human affairs, of course, has long been regarded as a

central feature of modern society, gaining particular salience during the period of the

Enlightenment. In the early seventeenth century, Francis Bacon was influential in challenging

authority and tradition and emphasising the importance of experimentation and inductive

reasoning to derive scientific knowledge capable of providing the foundation for efforts to

improve human welfare (Zagorin, 1998). The philosophical basis of this challenge wasstrengthened by Kants notion of . . . a pure reason, held universally by all rational beings . . .

employed to bring improvementto human society(Trigg, 2001, p. 222). Newtons laws of

motion provided an exemplar of the power of the new science, promising control over nature

and providing the basis for optimism about social and political progress (Bronk, 1998).

In spite of many challenges, most recently from post-modernists, this basic optimism about

the role of scientific knowledge remains embedded in Western liberal democratic political

systems. In the 1960s, the social sciences became an institutionalised component of

government policy-making, being seen as a key basis for more effective policy (Wagenaar,

1982). Indeed, Martin Bulmer (1987, p. 349) has argued that there was considerable over-

optimism at this time about the potential role of social science based upon . . . an almost

euphoric sense that social science really could change the world.This optimism was shaken

in the 1970s and 1980s by a growing scepticism based upon a lack of evidence that social

science research was actually influencing policy decisions. This scepticism was reinforced by

the Thatcher Governments stance, founded upon Sir Keith Josephs famous antipathy to social

science, characterised by Bulmer (1987) as . . . a general hostility to the social sciences and

social research, with tinges of philistinism . . ..

Nevertheless, it is doubtful whether this scepticism about the socialsciences inflicted any

lasting damage on the underlying faith in the scientific enterprise in the broader sense. Social

scientists were meanwhile busy seeking to rescue the notion of evidence-based policy through

work that indicated that the influence of research was much more complex than previously

supposed. Carol Weisss work in this area has been highly influential in highlighting thelimitations of instrumental notions of research use and emphasising its broader enlight-

enmentfunction, influencing the conceptualisation of issues, the range of options considered

and challenging taken-for-granted assumptions about appropriate goals and activities (Weiss,

1982). Based upon this work social scientists could proceed, comfortable in the knowledge

that while their research might not have an immediate direct impact on policy decisions, it

would nevertheless percolateinto the policy arena through a range of informal routes and

have a longer-term influence.

In the resurgence of enthusiasm for EBP that has accompanied the accession to power of

New Labour, such notions of enlightenment and conceptual functions of research have been

8/10/2019 S08 - Sanderson, 2003

5/16


overtaken by a renewed optimism about achieving more direct and instrumental use of

research in policy-making processes. These were ushered in by the Prime Ministerial

declaration that what counts is what works (Powell, 1999, p. 23). In this context, what

works can be taken as referring essentially to types of government intervention that are

effective in addressing the problems at which they are directed and achieving their intended

outcomes and effects. The Governments preoccupation with EBP has developed in the

context of a model of modern, professional policy makingthat has been propounded by theCabinet Office:

Good quality policy-making depends on high quality information and evidence. Modern

policy-making calls for the need to improve Departmentscapacity to make best use of

evidence, and the need to improve the accessibility of evidence available to policy-makers

(Bullock et al., 2001, p. 25).

Modern policy-making lies at the heart of the modernising governmentagenda, which

is seeking to make government more responsive and effective in achieving results (Cabinet

Office, 1999; Sanderson, 2001, 2002). The emphasis is very much on results, expressed in theform of measurable targets in government departmentsPublic Service Agreements (PSAs)

with the Treasury (HM Treasury, 2000). In a strong performance management regime,

departments are accountable for achievement against their targets and considerable emphasis

is given to evaluation . . . showing what worked well in improving public services and why,

and considering what further practical steps were needed to enhance service delivery and

improve effectiveness (National Audit Office, 2001). This is indeed now a familiar plea

indicating the predominance of an instrumentalview of evaluation in EBP.

The underpinning rationale of the Governments position on evidence-based policy-

making was articulated a couple of years ago by David Blunkett, then Secretary of State for

Education and Employment, in a much-quoted lecture to the Economic and Social Research

Council (ESRC) (DfEE, 2000). He argued that . . . rational thought is impossible without

good evidence . . . social science research is central to the development and evaluation of

policy(DfEE, 2000, p. 24); emphasised the Governments . . . clear commitment that we will

be guided not by dogma but by an open-minded approach to understanding what works and

why . . .(DfEE, 2000, p. 2); and expressed his passionate beliefthat . . . having ready access

to the lessons from high quality research can and must vastly improve the quality and

sensitivity of the complex and often constrained decisions we, as politicians, have to make

(DfEE, 2000, p.4).

In this rationalist vision, there is little sympathy for the broader enlightenment function

of social research, much of which is seen as . . . inward looking, too piecemeal . . . too

supplier-drivenrather than focusing on the key issues of concern to policy makers . . .(DfEE, 2000, p.8). Blunkett voiced concerns about the focus, relevance and timeliness of

research and indicated a strong desire for more research that is directly accessible, intelligible

and relevant to users in the policy community. The emphasis, therefore, is on enhancing

instrumentaluse. Whilst acknowledging the place for fundamental blue-skies research, he

placed the major emphasis upon research with practical applications. He highlighted the

need for research that . . . leads to a coherent picture of how society works: what are the

main forces at work and which of these can be influenced by government . . .; the need

. . . to be able to measure the size of the effect of A on B . . .; and the . . . huge potential

for quantitative analysis. . .(DfEE, p.22).

8/10/2019 S08 - Sanderson, 2003

6/16

Is it what worksthat matters? 335

It is easy to be cynical about politicians advocating the use of evidence in decision-making

when so much policy remains driven by the exigencies of the political process. However, if

we accept his arguments at face value, he can be seen as tapping into a rich seam of received

wisdom; as Robert Walker (2001, p. 307) argues, . . . few would quibble with Blunketts

aspiration to make social science research evidence central to the development and evaluation

of policy . . .Thus, on one level, it is self-evident that ideologically-driven policies potentially

represent a waste of public resources and that there is indeed a strong ethical case for ensuringthat scarce resources are directed into those policies that the evidence suggests are likely to be

most effective in addressing social problems (Judge and Bauld, 2001).

However, there are those who argue that the relationship between research evidence and

better policy decisions is not so self-evident. There are two main grounds for criticism. The

first questions thefeasibility of the projectto what extent can evidence provide the basis for

better policy? The second questions its desirabilityto what extent shouldpolicy be guided by

evidence? In the next section I will address the feasibility issue in a discussion of some aspects

of the role of evaluation in EBP; in the following section I will turn to the desirability issue

in a discussion of how much what worksdoes matter.

HOW MUCH CAN EVALUATION TELL US ABOUT WHAT WORKS?

It is clear, then that evaluation is seen as playing a major role in providing the required robust

evidence of what works. To what extent can evaluation deliver on this requirement? Of

course, this question raises fundamental issues in the philosophy and methodology of social

science to which I cannot do justice here. We could quickly become immersed in the

paradigm warsbetween positivists, social constructionists and realists and in this context we

cannot go there. I propose more modestly to focus on the current emphasis in the policy

evaluation community on theory-based evaluation as a means of deriving knowledge of causal

relationships which are seen as the key to understanding how and why policy initiatives work

in producing desired or intended social change (Davies et al., 1999; Walker, 2001).

It would appear that theory-based evaluation (TBE) has gained considerably increased

currency through its perceived contribution to EBP. The role of evaluation generally in this

context has been considered recently by the National Audit Office (2001, p. 25), which argues

that the appropriate response to increasing complexity in the policy making environment is

. . . to apply more powerful tools and draw on more specialist knowledge to enhance . . .

capacity to design and implement successful policies. Reliable and comprehensive

informationand sound analysisis crucial to the understanding of problems and the need for

policy intervention and . . . helps to establish what works and to identify optimumopportunities for intervention (National Audit Office, 2001, p. 67, 26). Considerable

emphasis is given in this model of modern policy making to the role of evaluation in learning

lessons:

Evaluation is important for determining the extent to which a policy has met or is meeting

its objectives and that those intended to benefit have done so. Evaluation can also help

departments learn lessons and share good practice in policy design and implementation.

For long-term policies, evaluation can identify ways in which the policy can be improved

or developed to increase its impact (National Audit Office, 2001, p. 14).

8/10/2019 S08 - Sanderson, 2003

7/16


In their review of policy-making in various government departments, the NAO

highlighted a need for evaluation to be more practical, . . . showing what worked well in

improving public services and why, and considering what further practical steps were needed

to enhance service delivery and improve effectiveness(National Audit Office, 2001). This is

indeed now a familiar plea indicating the predominance of an instrumental view of

evaluation.

In the context of this agenda, considerable attention is now being focused on the promiseof TBE to deliver knowledge of the causal relationships that are seen as the key to

understanding what works. Recent work has emphasised the importance of improving our

theoretical knowledge of how and why different approaches to policy intervention workin

producing desired or intended social change (Davies et al., 2000; Walker, 2001). Pawson and

Tilley (1997) established the new mantra of evaluatorswhat works, for whom, in what

circumstances, and whywhich derives from the realist concern to understand how policy

mechanisms operate in different contextual circumstances to change outcomes. Of course,

this resonates well with the Governments vision of EBP. In this context there has been

increasing criticism of the instrumental rationalityof goal-oriented and method-oriented

evaluation, which de-contextualises evaluation, restricts the analysis of potential effects, andfails to provide an understanding of how interventions produce such effects (Chen, 1990).

Therefore, TBE is seen as a primary means of generating the causal knowledge required for

more effective policy design and implementation by testing the validity of the causal

assumptions upon which policy initiatives and programmes are based (Weiss, 1995). However,

while undoubtedly holding out considerable promise for the development of knowledge to

underpin more effective policy-making, TBE presents significant challenges in terms of

articulating theoretical assumptions and hypotheses, measuring changes and effects,

developing appropriate tests of assumptions and hypotheses, and in terms of the

generalisability of results obtained in particular contexts (Weiss, 1995). It is not appropriate to

discuss the technical arguments here (see Connell et al., 1995; Fulbright-Anderson et al.,

1998; Rogers et al., 2000) but I do want to say something about causal knowledge since this

is central to the promise of TBE.

Even if we set aside the potential difficulties that can arise due to the specification and

empirical measurement of theoretical constructs, substantial problems surround the process of

theory testing such that even in relatively simple policy interventions it is extremely difficult

to derive robust causal attribution. Even such simple interventions can involve a large number

of links in the theoretical chaininterlinked assumptions about how they will achieve their

intended effectsand in practice evaluations usually either have to be highly selective in the

analysis of causal links or have to parcel them up(Weiss, 2000). Scriven (1998), makes the

point that evaluators are not usually in the business of seeking deep explanations. In practice,

the causal linkages between the observed intervention and the observed effect will remainsomething of a black boxand our capacity as evaluators to open this up and identify how the

intervention works in terms of causal mechanisms is quite limited, especially when

interventions become more complex (Granger, 1998). Many of our key social problems

require such complex, cross-cuttinginterventions.

Realists argue that it is possible to derive causal understanding through direct analysis of

how theorised mechanisms work in particular contexts to produce outcomes (Pawson and

Tilley, 1997). Experimentalists counter that this approach does not provide a valid

counterfactual, which is required for robust inference that the outcomes are genuine

products of the intervention and would not have occurred in its absence (Cook, 2000).

8/10/2019 S08 - Sanderson, 2003

8/16


However, while experiments can provide a basis for causal inference in the limited

circumstances where a uniform, standardised intervention is provided for a clearly defined

target group under tightly-controlled conditions, this is based solely on the successionist

assumption that effect must follow the intervention if all other possible causes are controlled

out. They do not provide an understanding of how the effects are produced thus creating

problems of external validitythe ability to generalise from the result to other situations.

Although there remain matters of philosophical dispute, I would line up behind advocatesof the need to use the full range of methodological approaches to provide the best hope of

capturing various facets of policy interventions and piecing together a picture of how they

produce change in particular contextual circumstances (Sanderson, 2000). This is a stance of

very modest expectations in contrast to the bullishclaims of some proponents of TBE made

in the context of rationalist conceptions of evidence-based policy-making. Even Carol Weiss

(1997) refers to the aggressively rationalistic stanceof TBE but her overall position represents

a cautious and pragmatic response seeking the middle way between the horns of the Cartesian

dilemma. It is worth quoting her at length:

In my most optimistic moments, I succumb to the notion that evaluations may be able topin down which links in which theories are generally supported by evidence and that

program designers can make use of such understanding in modifying current programs and

planning new ones . . . Such hopes are no doubt too sunny. Given the astronomical variety

of implementations of even one basic program model, the variety of staffs, clients,

organizational contexts, social and political environments, and funding levels, any hope for

deriving generalizable findings is romantic. Nevertheless, theory-based evaluation can add

to knowledge. Even relatively small increments of knowledge about how and why

programmes work or fail cannot help but improve program effectiveness. And that is what

program evaluation is all about (Weiss, 2000, p. 44).

However, not all reactions to this problem are so pragmatic. I think we can legitimately

question whether the use of public money in resource-intensive theory-based evaluations is

appropriate if they are producing relatively small increments of knowledge. Improving the

knowledge base for policy intervention is clearly important but has to be set against other calls

on scarce public resources in terms of the value added for society. It will be interesting to see

how this issue plays out as we get more experience with TBE and as it becomes clearer just

how much it can deliver robust evidence of what works.

Of course, in the extreme, postmodernists will reject this whole agenda of attempting to

enhance the role of reason in the guidance of human affairs but many who acknowledge

the need to strive for social improvement reject such a nihilistic position (Trigg, 2001;

Oakley, 2000). Nevertheless, some critics take issue with the technical-rationalistorientation represented by TBE. This stance is represented by Thomas Schwandt, who

regards evaluation in its dominant guise essentially as modernist project designed . . . to

tame the unruly social world, to bring order to our way of thinking about what does and

does not work for improving social life(Schwandt, 2000a). In this project, rationality is a

matter of correct procedure or method in a context where . . . policymakers seek to

manage economic and social affairs rationallyin an apolitical, scientized manner such that

social policy is more or less an exercise in social technology(Schwandt, 1997, p. 74). He

sees the problem in the domination of technical rationality and expertise, which subsumes

issues of moral-political judgement. What he finds objectionableis the belief that practice

8/10/2019 S08 - Sanderson, 2003

9/16


will somehow be less ambiguous or more rational if we can only find the right ways to

generate and apply evaluation knowledge (Schwandt, 2000a). He argues that the application

of scientific method to contemporary life can lead to the deformation of praxis, and

maintains that social scientific knowledge (which is general and theoretical) cannot provide

the primary basis for evaluation judgements under conditions of plurality, uncertainty and

difference(Schwandt, 2000b).

While acknowledging the role to be played by such knowledge, Schwandt argues thatevaluators must recognise the need for practical knowledge or wisdomto help practitioners

understand their practice better and to make wise moral-politicaljudgements:

While of course we would like to have at our disposal the best general, scientific

knowledge we can acquire, the corrigibility, ambiguity, and circumstantiality of everyday

evaluative judgement cannot be eliminated, replaced or refined by relying on scientific

method and its associated rationality (Schwandt, 2000b).

Central to Schwandts perspective on evaluation is the notion of critical intelligence:

Critical intelligence . . . is the ability to question whether the . . . (end) . . . is worth

getting to. It requires not simply knowledge of effects, strategies, procedures and the like

but the willingness and capacity to debate the value of various ends of a practice. It requires

acknowledging and understanding the force of tradition (prejudices) in shaping the

conceptualisation of those ends, the means used to frame those ends, and the practices

employed to assess their effects and a simultaneous effort to transform that knowledge in

the process of coming to that understanding. This is fundamentally an exercise in practical-

moral reasoning. (Schwandt, 1997, p. 79.)

From this perspective, what matters is not so much what worksas what is appropriatein

particular circumstances and evaluation is not merely techniqueinvolving robust objective

analysis but rather more craft activity involving reasoned judgement of various forms of

knowledge and normative implications. This leads us, therefore, to the issue of the desirability

of basing policy decisions on research evidenceto question just how much what works

does matter.

HOW MUCH DOES WHAT WORKSMATTER?

Schwandts conception of evaluation is consistent with analyses of the broader policy-making

process as more a communicativeprocess based upon dialogue and argumentation than atechnicalprocess based upon evidence. A key work in this tradition is by Giandomenico

Majone (1989) who takes issue with the . . . rationalist fallacy of believing that theories and

ideas alone are powerful enough to determine the course of events, and of interpreting policy-

making as a purely intellectual exercise . . .(Majone, 1989, p.146) and criticises decisionism

as neglecting the role of social processes and rules. He presents an alternative conception of

policy making as a process of deliberation and argumentation, drawing on Aristotle for his

view of policy analysis as craft work, and emphasising the role of argument. . . a complex

blend of factual statements, interpretations, opinions, and evaluations(Majone, 1989, p.63).

In this perspective, the moral/ethical dimension figures prominently:

8/10/2019 S08 - Sanderson, 2003

10/16


. . . the choice of policy instrument is not a technical problem that can be safely delegated

to experts. It raises institutional, social and moral issues that must be clarified through a

process of public deliberation and resolved by political means (Majone, 1989, p. 143).

In Majones view, the notion of EBP places too much emphasis on the potential role of

causal knowledge in improving policy effectiveness and insufficient emphasis on the

normative, institutional and organisational context in which decisions and choices are madeand action is taken.

Alasdair MacIntyre (1984) also draws on Aristotle in his critique of the notion that rational

action is an essentially technical matter of applying morally neutral expertise as the means for

achieving authoritative ends. He argues that this notion neglects the intr insic virtuessuch

standards as fairness, truthfulness, trust and honestythat are embodied in human practices

and which give such practices an inherently moral and ethical character. He defines a practice

as a cooperative human activity that has its own standards of excellence. In striving to achieve

these standards, practitioners realise internal goods, which are satisfactions intrinsic to the

activity derived from doing it well. A virtue, then, is . . . an acquired human quality the

possession and exercise of which tends to enable us to achieve those goods which are internalto practices . . .(MacIntyre, 1984, p. 191). The implication of this position is that the model

of rationality underpinning evaluation should not be based upon an instrumentalnotion of

the effectiveness of means to given ends but rather upon a practicalnotion of the appropriateness

of action from a broader ethical-moral standpoint.

The philosophical basis for such a position is traceable back to Aristotle and the pre-

Enlightenment acceptance that scientific reasoning and expertise was but one amongst many

legitimate bases for belief and actiona situation that Stephen Toulmin (2001, p. 29) terms

the Balance of Reason. Toulmin argues that this balance was disrupted in the seventeenth

century as, under the influence of Galileo and Descartes, exact sciences susceptible to

mathematical methods, theoretical abstraction and logical deduction gained ascendancy as

apparent means of overcoming the uncertainties and ambiguities that had previously been

accepted. Thus, began what John Dewey called the quest for certainty, which culminated in

Newtonian physics being taken as showing that the Solar System was an exemplar of a

rationally intelligible systemdemonstrating . . . regularity, uniformity, and above all stability

(Toulmin, 2001, p. 48). Toulmin argues that the hegemony of Newtonian dynamics, bolstered

by its . . . intellectual coherence with a respectable picture of Gods Material Creation . . .

(Toulmin, 2001, p.79), provided social sciences with an unrealistic and, indeed, distorting

exemplar of Serious Science, which has blinded them to matters of political and moral

concern and practical relevance:

. . . (A) traditional reliance on Euclidian and Newtonian models of theory continues tofocus attention on doing your sums right and conceals the equally important task of

making sure that you are doing the right sums; in other words, doing calculations that are

directly relevant to the practical situation in question (Toulmin, 2001, p. 66).

Toulmin calls for the restoration of the Balance of Reason by acknowledging the validity

and role of practical wisdomin assessing what is reasonableor appropriatein dealing with

human and social problems. Practical wisdom is a translation of Aristotles concept ofphronesis

which, in contrast to episteme(theoretical knowledge) and techne(instrumental knowledge),

involves, according to Dryzek (1990, p. 9) . . . persuasion, reflection upon values, prudential

8/10/2019 S08 - Sanderson, 2003

11/16


judgement, and free disclosure of ones ideas.Aristotles discussion of phronesis referred to

medical practice and helmsmanship, realms of skilled and experienced practice that rely

heavily on tacit knowledge. Such knowledge is grounded in experience but largely taken for

granted and embedded in routines that go without saying; it is not easily conveyed through

written reports and papers. A closely related concept is that of metis, which translates from

Greek as knack, wit, or cunning. Toulmin argues for the restoration of these forms of tacit

knowledge in a Balance of Reason, which recognises that . . . all scientific knowledge is abalance of the theoretical with the practical, the verbal with the non-verbal(Toulmin, 2001,

p. 183).

I would argue that this position provides a more robust basis for dealing with the major

human and social problems that we face than instrumental forms of rationality that have

become embedded in Western liberal democratic political systems. A major challenge derives

from the increasing complexity of late modernsociety under the influence of increasingly

globalised processes of economic and technological change (Giddens, 1990). According to

Smart (1999, p. 63) We find ourselves abroad in a world in which social theory and analysis

is no longer able, with any credibility, to provide a warrant for political practice and ethical

decision-making. However, the dilemma is that . . . questions concerning politicalresponsibility and ethical decision-making, the difficulties of adjudicating between the

expression and pursuit of self-interest and the promotion and adequate provision of the public

domain, as well as the problems encountered in everyday social life of making a choice or

taking a stand, have if anything become analytically more significant . . .(Smart, 1999).

Dryzek (1990) argues that dominant responses to the challenge of complexity tend to

reflect the preoccupations of instrumental rationality, emphasising forms of analysis and

technical aides to decision making that are more sophisticated, in order to maintain control

over a problematical environment. We have seen aspects of just such responses in the

Governments approach to EBP in the context of the performance management control-

strategy. From the perspective of practical reason, such responses will have limited success for

two main reasons. First, by focusing on formal scientific and technical knowledge, they

neglect the key role played in problem solving by practical wisdom and informal tacit

knowledge. Second, by conceiving of rationality in terms of means to given ends, they neglect

the ethical-moral dimension of problem solving. Indeed, these two aspects should be seen as

inter-related in that practical knowledge or wisdom is necessarily applied in practice within

a normative framework.

We can consider this further by looking briefly at the organisational context in which

professional practice occurs in relation to policy-making and public service delivery. Recent

research on organisations has emphasised the importance of social relations and informal

processes founded upon tacit knowledge (Hatch, 1997; Moingeon and Edmondson, 1996).

New institutionalismalso highlights the role of the informal normative orderdefined interms of norms, routines and conventions, which are largely tacit and implicit and deeply

ingrained in organisational life (March and Olsen, 1989; Lowndes, 1997). March and Olsen

(1989) argue that the focus should therefore be on the appropriateness of action and

behaviour, defined in relation to the normative order of obligation and necessity, rather than

on the rational orderof preference and calculation and consequence.

Given this conception of the organisational and institutional context in which policy-

makers and practitioners encounter evidence, make judgements and take decisions, rational-

decisionisticmodels appear highly simplistic and distorting. For example, with reference to

evidence-based practice in social work, Stephen Webb (2001) argues that professionals

8/10/2019 S08 - Sanderson, 2003

12/16


working in public sector organisations employ reasoning strategies which . . . consistently fail

to respect the canons of rationality assumed by the evidence-based approach(Webb, 2001, p.

64). People employ cognitive heuristics that are selective in relation to evidence and decisions

are influenced by factors such as the politics of inter-agency relations and internal

organisational interest groups and . . . are based upon the reflexive understanding of

contestable beliefs and meanings and not determinate judgements(Webb, 2001, p. 68).

The role of the normative context of social and political rules, conventions and structuresof power and authority in shaping professional practice and action is emphasised in John

Foresters (1993) analysis of planning practice as communicative action. Not only does such

practice draw on sources of knowledge other than explicit scientificevidence (in particular

on tacit and experiential practitioner knowledge) but it also necessarily addresses normative

considerations: . . . questions of norms and values are . . . necessarily influential and

constitutive of the very sense of action itself (Forrester, 1993, p. 72). In the context of

community care, Janet Lewis (2001) has recently highlighted the danger, in evidence-based

practice, of excluding both the practical wisdomof the experienced practitioner and the

experience of service users as forms of knowledge that do not constitute valid evidence.

However, the problem here is not limited to the restriction of the cognitive basis of practice.The exclusion of these perspectives also excludes the value stances that go with them,

resulting in a de facto privileging of the normative commitments of academics, managers and

policy makers (cf. Beresford, 2001).

The argument can be applied to professionals working in other public service contexts.

Scientific evidence tends to be at a relatively high level of generality whereas professionals are

faced with decisions about dealing with particular problems in particular circumstances and

institutional contexts. Research evidence is notoriously slipperyeven in terms of informing

general policy guidelines to deal with such problems. When it comes to the specifics of

practice, it recedes even further into the background in decisions on appropriateaction. To

take an example from the educational field, a schools policy for dealing with bullying should

certainly be informed by evidence of what is generally effectivebut responses to bullying

incidents will be dominated by practice wisdom, cautiously teasing out the most appropriate

course of action in the specific circumstances in a context of informal rules, heuristics, norms

and values. The question for teachers is not simply what is effectivebut rather, more broadly

it is, what is appropriate for these children in these circumstances.

CONCLUSION: WHAT DOES MATTER?

I would not want my argument to be misconstrued. I certainly do not line up with post-

modernists who take a nihilistic view of a world beyond the reach of redemptive politics(touse OSullivans phrase again). I have long believed in the need to apply the findings from

social scientific research more systematically in our collective efforts to address social problems

and improve the lives of those suffering poverty and disadvantage. The danger lies in the

instrumental conception of rationality that underpins what Frank Fischer (1990) terms

technocratic politicsand what Thomas Schwandt sees as the reduction of policy-making to

an exercise in social technology.

In this context, I have outlined concerns over the current wave of optimism about EBP

and about the potential of theory-based evaluation. Although I would count myself as a

supporter of this approach to evaluation, I have doubts about the feasibility of its promise,

8/10/2019 S08 - Sanderson, 2003

13/16


especially in relation to complex, crosscutting, multi-intervention programmes. Such contexts

present very severe challenges to evaluators in seeking to derive evidence of what worksin

terms of an understanding of the causal mechanisms by which an intervention achieves

change and how the operation of such mechanisms is influenced by relevant contextual

factors. I suspect that much of the evidence produced by ostensibly theory-based evaluations

will continue to provide limited purchase on causal mechanisms and present serious problems

of external validity.In part, this is due to the character of the evaluation enterprise. Evaluation is, by its nature,

part of a policy process conditioned by interests located in social and political processes

operating in particular organisational contexts. Thus, as an exercise in practical-moral

reasoning, evaluation takes place within an institutional framework, which differs in

significant ways from that which is applicable to the pursuit of pure social scientific

knowledge. For example, the policy client normally has a strong interest in seeing findings

that can be used to improve policy design or implementation within timescales dictated by the

political process. Evaluation funded by government will inevitably be focused to a significant

degree on a process of learning and improvement rooted in short-term practice rather than

one oriented to longer-term development of research evidence. Evaluators must satisfy theneeds and demands of multiple stakeholders, often within limited budgets, and this involves

balancing and compromising amongst various prioritiesa strategy not particularly

conducive to the kind of sustained, in-depth research that Weiss (1997) argues is necessary for

successful theory-based evaluation.

Ultimately, I think we have to be modest in our expectations about the kind of knowledge

that we can derive as a basis for policy-making. We have seen that in his concern to promote

knowledge for human good, Stephen Toulmin urges us to . . . recall the practical wisdom of

the sixteenth century humanists, who hoped to recapture the modesty that had made it

possible for them to live happily with uncertainty, ambiguity and pluralism(Toulmin, 2001,

p. 80). Dupre (2001) advocates a thorough-going pluralism that . . . goes all the way down

to the basic metaphysical issues of causality and what kind of things there are(Dupre, 2001,

p. 183). He argues that there is no one fundamental perspective that can explain human

behaviour and, indeed, that . . . it remains easier to say what does not work than what does

(Dupre, 2001, p. 184). We can achieve fragments of the whole picture from various

perspectives and we need to develop skills of integrating these fragments; . . . trying to make

one or even a few such fragments stand for the whole presents us with a deformed image of

ourselves(Dupre, 2001, p. 183).

Moreover, we have seen how practical wisdom must rely heavily on tacit knowledge

grounded in experience and how the application of such knowledge in professional practice

necessarily occurs within a normative framework of values, rules, heuristics and conventions

that can make decision-making and action as much a matter of obligation and necessity as amatter of rational analysis. For Stephen Toulmin, practical wisdom helps us . . . in untying the

knots in which our lives enmesh us(Toulmin, 2001, p. 123) and in this task . . . intelligent

analysis of the factual soil from which our problems spring . . .must be . . . guided by ideals

that make rational assessments stepping stones to reasonable decisions (Toulmin, 2001, p.

213). This is in accordance with John Deweys notion of creative intelligencethrough which

we can transform the world for the better. Such intelligence has relevance to our ends and

values and . . . frees action from a mechanically instrumental character . . .; its function is . . .

to project new and more complex endsto free experience from routine . . . to liberate and

liberalize action(Dewey, 1993, p. 67). From this perspective, what matters is arriving at such

8/10/2019 S08 - Sanderson, 2003

14/16


reasonabledecisions and ensuring that we take action that is appropriateto situations that

are both morally and factually ambiguous (cf. Harmon, 1995). The practical rationalityof

this process requires inclusive, open and free debate, ensuring that all those with a stakein

decisions are able to bring to bear their knowledge (in whatever form) andtheir normative

commitments.

If we accept this position, and acknowledge the need to nurture the application of practical

wisdom in our efforts to improve the world, then we need to broaden the focus of evaluationbeyond the technicalconcerns of measuring effects, identifying causes and assessing what

works. It can help in the practical task of identifying what is appropriate or reasonable. To do

this, it must be acknowledged that the ethical and moral implications of policies and the values

and goods(and bads) that they promote are amenable to rationalconsideration and debate

(cf. Julnes et al., 1998). Broadening the focus of evaluation in this way also involves broadening

its methodologies beyond analytical techniques to include methods and accompanying

institutional frameworks to promote full, free and open normative debate among all those

with a stake in the policies concerned, including service users and citizens.

In this way, evaluation can strengthen the basis for making wise policy choices with

profound ethical and moral implications. It can strengthen our capacity to answer whatZagorin (1998, p. 224) calls Tolstoys anguished questionwhat shall we do and how shall

we live?It is this question that tasked John Dewey, who was passionately committed to the

application of intelligence to the solution of human and social problems and transforming the

world for the better, advocating experimentation and an open-minded will to learn. His is

the appropriate final word in advocating . . .

. . . the necessity of a deliberate control of policies by the method of intelligence, an

intelligence which is not the faculty of intellect honored in text-books and neglected

elsewhere, but which is the sum-total of impulses, habits, emotions, records, and

discoveries which forecast what is desirable and undesirable in future possibilities, and

which contrive ingeniously on behalf of an imagined good (Dewey, 1993, p. 9).

REFERENCES

BERESFORD, P. (2001). Evidence-based care: developing the discussion, Managing Community Care,

9, 36.

BRONK, R. (1998). Progress and the Invisible Hand: The Philosophy and Economics of Human Advance.

London: Warner Books.

BULLOCK, H., MOUNTFORD, J. and STANLEY, R. (2001). Better Policy Making. Centre for

Management and Policy Making, London: Cabinet Office.

BULMER, M. (Ed.) (1987). Social Science Research and Government: Comparative Essays on Britain and theUnited States. Cambridge: Cambridge University Press.

CABINET OFFICE (1999). Modernising Government. Cm. 4310. London: The Stationery Office.

CHEN, H-T. (1990). Theory-Driven Evaluations. Newbury Park: Sage Publications.

CONNELL, J. P., KUBISCH, A. C., SCHORR, L. B. and WEISS, C. H. (Eds) (1995). New Approaches

to Evaluating Community Initiatives, Volume 1: Concepts, Methods and Contexts. Washington, DC: Aspen

Institute.

COOK, T. D. (2000). The False Choice Between Theory-Based Evaluation and Experimentalism. In:

ROGERS, P. J., PETROSINO, A. J., HEUBNER, T. A. and HASCI, T. A. (Eds) Program Theory

Evaluation: Practice, Promise, and Problems, New Directions in Evaluation, No. 87, 2734. San Francisco:

Jossey Bass.

8/10/2019 S08 - Sanderson, 2003

15/16


DAVIES, H. T. O., NUTLEY, S. M. and SMITH, P. C. (1999). Editorial: what works? The role of

evidence in public sector policy and practice, Public Money and Management, 19, 35.

DAVIES, H. T. O., NUTLEY, S. M. and SMITH, P. C. (Eds) (2000). What Works? Evidence-Based Policy

and Practice in Public Services. Bristol: Policy Press.

DEWEY, J. (1957). Reconstruction in Philosophy. Enlarged Edition. Boston, MA: Beacon Press.

DEWEY, J. (1993). The Need for a Recovery of Philosophy. In: MORRIS, D. and SHAPIRO, I.

(Eds)John Dewey: The Political Writings. Indianapolis, IN: Hackett Publishing Co.DfEE. (2000). Influence or Irrelevance: Can Social Science Improve Government? Secretary of States ESRC

Lecture Speech, 2 February. London: Department for Education and Employment.

DRYZEK, J. S. (1990). Discursive Democracy: Politics, Policy and Political Science. Cambridge: Cambridge

University Press.

DUPRE, J. (2001). Human Nature and the Limits of Science. Oxford: Clarendon Press.

FISCHER, F. (1990). Technocracy and the Politics of Expertise. Newbury: Sage Publications.

FORESTER, J. (1993). Critical Theory, Public Policy and Planning Practice: Toward Critical Pragmatism. New

York: State University of New York Press.

FULBRIGHT-ANDERSON, K., KUBISCH, A. C. and CONNELL, J. P. (Eds) (1998). New

Approaches to Evaluating Community Initiatives, Volume 2: Theory, Measurement and Analysis.Washington,

DC: Aspen Institute.

GIDDENS, A. (1990). The Consequences of Modernity. Cambridge: Polity Press.

GRANGER, R. C. (1998). Establishing Causality in Evaluations of Comprehensive Community

Initiatives. In: FULBRIGHT-ANDERSON, K., KUBISCH, A. C. and CONNELL, J. P. (Eds)

New Approaches to Evaluating Community Initiatives, Volume 2: Theory, Measurement and Analysis.

Washington DC: Aspen Institute.

HATCH, M. J. (1997). Organization Theory: Modern, Symbolic, and Postmodern Perspectives. Oxford:

Oxford University Press.

HARMON, M. M. (1995). Responsibility as Paradox: A Critique of Rational Discourse on Government.

Thousand Oaks, CA: Sage.

HM TREASURY (2000). 2000 Spending Review: Public Service Agreements. Cm. 4808. London: HM

Treasury.

JUDGE, K. and BAULD, L. (2001). Strong theory, flexible methods: evaluating complex community-based initiatives, Critical Public Health, 11, 1938.

JULNES, G., MARK, M. M. and HENRY, G. T. (1998). Promoting realism in evaluation: realistic

evaluation and the broader context, Evaluation, 4, 483504.

LEWIS, J. (2001). What works in community care?, Managing Community Care, 9, 36.

LOWNDES, V. (1997). Change in public service management: new institutions and new managerial

regimes, Local Government Studies, 23, 4266.

MACINTYRE, A. (1984).After Virtue: A Study in Moral Theory. Notre Dame: University of Notre

Dame, IN Press.

MAJONE, G. (1989). Evidence, Argument and Persuasion in the Policy Process. New Haven, CT: Yale

University Press.

MARCH, J. G. and OLSEN, J. P. (1989). Rediscovering Institutions: The Organizational Basis of Politics.

New York: The Free Press.

MOINGEON, B. and EDMONDSON, A. (Eds) (1996). Organisational Learning and Competitive

Advantage. London: Sage.

NATIONAL AUDIT OFFICE. (2001). Modern Policy-Making: Ensuring Policies Deliver Value for Money.

London: NAO.

OAKLEY, A. (2000). Experiments in Knowing: Gender and Method in the Social Sciences. Cambridge: Polity

Press.

OSULLIVAN, N. (1993). Political integration, the limited state, and the philosophy of postmodern-

ism, Political Studies, XLI, 2142.

PAWSON, R. and TILLEY, N. (1997). Realistic Evaluation. London: Sage Publications.

8/10/2019 S08 - Sanderson, 2003

16/16

Documents

S08 - Sanderson, 2003