Embed Size (px)
Citation preview
The ‘heart’ of evaluation influence:a compelling evaluative argument
AES Conference 2011
Heather Nunns
Workshop overview
• Why evaluative reasoning and argument are important
• Defining evaluative reasoning • The origins of evaluative reasoning• ‘Unpacking’ evaluative reasoning• Connecting evaluative reasoning and
argument
My contentionNo matter ...the sophistication of our evaluation design ...our attention to methodological rigor ...our strategies and tools to encourage evaluation use,
an evaluation will lack impact without sound evaluative reasoning and argument
Evaluative reasoning defined
“The systematic means for arriving at evaluative conclusions, the principles that support inferences drawn by evaluators” Fournier, 1995, p.1
“...professionalised evaluation has spent much of its time and effort on developing methodological sophistication and less so on logical sophistication. Understanding the reasoning process used to establish evaluative conclusions ...has to be the field’s greatest unmet challenge” Fournier, 1995, p.1.
Values
Value (noun): – mental pictures of idealised state of affairs or models of
idealised behaviour considered by a person to be of worth
Value (verb): – the act of making a reflective judgment of worth (as in
‘valuing’)– to value something is to take up a certain positive attitude
towards it Magendanz, 2003
Values
Values are at the heart of evaluation
Evaluation = judgment of the merit (quality), worth (value), significance (importance)
of an evaluandHouse 1999, Scriven 1991, Stufflebeam 2001
Evaluators are in the business of assessing value of some sort:“How responsive is the programme delivery..?”
“Is X or Y the more effective approach...?”“Which parts of the programme are worth replicating...?”
Western philosophy: the traditional fact – value distinction
FACTS
VALUES
The origins of evaluative reasoning
• Axiology = a branch of western philosophy focused on the study of values (early 20th century)
• Asserted that it is possible to draw ‘logical’, objective conclusions about values Richard Hare (1919-2002), Paul W. Taylor (?), Nicholas Rescher (1928 - )
How to assess value Hare 1967, Rescher 1969, Taylor 1961
1. Identify the object (X) and the value to be applied to the object
2. Identify the class of comparison to which X belongs (Z)
3. Identify norms for Z
4. Develop a set of ‘operational’ statements describing levels of performance for each of the norms of Z
5. Determine the characteristic(s) that X has
6. Compare X’s characteristics with the operational statements to come to an evaluative conclusion
7. Justify the norms and standards used
Michael Scriven
Evaluation Thesaurus, 1991
Reasoning, 1976
Over 100 publications about evaluation in 44 years
General logic of evaluation Scriven 1967, 1980, 1991, 1994
Making an evaluative judgment involves:
1.establishing criteria of merit for the evaluand
2.constructing standards for the criteria
3.measuring performance of the evaluand against the criteria
4.synthesising and integrating data into a judgment of merit or worth
Hare (1967), Rescher (1969), Taylor (1961) Scriven’s logic of evaluation (1967, 1980, 1991, 1994)
• Identify the object (X) and the value to be applied to the object
• Establish criteria of merit for the evaluand
• Identify the “class of comparison” to which X belongs (Z)
• Identify norms for Z • Construct standards for the criteria • Develop a set of operational statements
describing levels of performance for each of the norms of Z
• Determine the characteristic(s) of X (the “good making characteristics”)
• Measure performance of the evaluand against the criteria
• Synthesise and integrate data into a judgment of merit or worth
• Compare X’s characteristics with the operational statements above to come to an evaluative conclusion
• Justify the norms used
Comparing evaluation approaches
EVALUATION (GENERAL) LOGIC Scriven
EVALUAND
Explanation
Working logic, Fournier 1995
How the general evaluation logic is applied in practice depends on the:
• phenomenon: what is being evaluated
• problem: what is the ‘problem’ being investigated
• question: what question(s) are being asked
• claim: what kind of claims(s) are being made
Product evaluation Programme evaluation Policy evaluation
1. Phenomenon A product A programme A policy intervention
2. ‘Problem’ Performance effectiveness
How well the implementation of the programme is working
Effectiveness of the policy intervention
3. Question(s) Is X a good one of its kind? Is X better/less better than others of its kind?Is X value for money?
Is the implementation working as intended?What improvements to process are required?
What are the outcome(s) of the policy intervention?How valuable are these outcomes to participants?
4. Evaluative claim
Performance and value
Process/implementation performance
Causal and value
Examples of working logics, Fournier 1995
EVALUATION (GENERAL) LOGIC Scriven
EVALUAND
WORKING LOGIC Fournier
Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)
Context
EVALUATION (GENERAL) LOGIC Scriven
EVALUAND
WORKING LOGIC Fournier
Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)
Context
+ VALUES criteria & standards identified
Thinking further about ‘valuing’
“Ethical discussion aims at making us more critically aware of what we are
doing. It brings us back to thinking about what it is to be a good evaluator,
and to ask in whose interests should we be acting and for what
purpose? These are ethical questions, and they should take precedence
over technical questions about how to do evaluation.”
Tom Schwandt, Evaluation Practice Reconsidered, 2002, p.154
Valuing approaches: Schwandt, 1997
• Analytical value-free approaches:Evaluators should not make value judgments. They can describe the value positions of stakeholders and participants but not make a judgment about which is best e.g. Shadish, Cook & Leviton, 1991
• Emancipatory value-committed approaches:Evaluation practice should challenge values that serve existing power inequalities and be linked to political action e.g. Mertens, 2010
• Value-critical approaches: Evaluators use their expertise to add to and encourage practitioners’ reflective, conversational critiques of the value commitments embedded in their practice e.g. Patton’s developmental approach
EVALUATION (GENERAL) LOGIC Scriven
EVALUAND
WORKING LOGIC Fournier
Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)
Context
+ VALUES criteria & standards identified
Valuing approaches(House & Howe, Schwandt, Shadish, Cook & Leviton, Stake)
EVALUATION (GENERAL) LOGIC Scriven
EVALUAND
WORKING LOGIC Fournier
Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)
Context
+ VALUES criteria & standards identified
Valuing approaches(House & Howe, Schwandt, Shadish, Cook & Leviton, Stake)
+ Evidence
Surprises, puzzles, comments, questions...
EVALUATION (GENERAL) LOGIC Scriven
EVALUAND
WORKING LOGIC Fournier
Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)
Context
+ VALUES criteria & standards identified
Valuing approaches(House & Howe; Schwandt; Shadish, Cook & Leviton; Stake)
+ Evidence
PROBATIVE INFERENCE Scriven
EVALUATIVE JUDGMENT
EVALUATION (GENERAL) LOGIC Scriven
EVALUAND
WORKING LOGIC Fournier
Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)
Context
+ VALUES criteria & standards identified
Valuing approaches(House & Howe, Schwandt, Shadish, Cook & Leviton, Stake)
+ Evidence
PROBATIVE INFERENCE Scriven
EVALUATIVE JUDGMENT
Evaluative argument House
Ernie HouseValues in evaluation and social research, 1999
Evaluating with validity, 2010
Professional Evaluation: Social impact and political consequences, 1993
House, 1977
“...evaluation persuades rather than convinces,
argues rather than demonstrates, is credible rather
than certain, is variably accepted rather than
compelling” p.6
Evaluation as argumentation:
Evaluative argument – why is it important?
“My concern is that in the press to master methods of generating data, we ignore the idea of developing a warranted argument – a clear chain of reasoning that connects the grounds, reasons or evidence to an evaluative conclusion” Schwandt, 2008.
“Worrying about warrant is a core evaluator responsibility. It is because our inferences are consequential that we must have confidence that they are warranted” Greene, 2011.
An evaluative argument
• Toulmin et al (1979) has identified a logic of reasoning underpinning all types
of inquiry that aims to build an argument. This logic consists of six main
features which include claims, evidence, warrants, backings.
• A warrant is the “because” part of an argument. It legitimates the inference
from the claim and evidence to the conclusion by appealing to an appropriate
authority.
• Backings are added authority as to why the warrant should be accepted as
legitimising the inference.
EVALUATION (GENERAL) LOGIC Scriven
EVALUAND
WORKING LOGIC Fournier
Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)
Context
+ VALUES criteria & standards identified
Valuing approaches(House & Howe, Schwandt, Shadish, Cook & Leviton, Stake)
+ Evidence
PROBATIVE INFERENCE Scriven
EVALUATIVE JUDGMENT
Warrants & backings Toulmin
Evaluative argument House
and finally...the evaluative judgment
Our end point: an evaluative judgment that is legitimate and justified (Fournier, 1995)
But !
the synthesis debate (House, Scriven, Stake)
EVALUATION (GENERAL) LOGIC Scriven
EVALUAND
WORKING LOGIC Fournier
Values pluralism (Greene, Kushner, Lincoln & Guba, Schwandt)
Context
+ VALUES criteria & standards identified
Valuing approaches(House & Howe, Schwandt, Shadish, Cook & Leviton, Stake)
+ Evidence
PROBATIVE INFERENCE Scriven
EVALUATIVE JUDGMENT
Warrants & backings Toulmin
Evaluative argument House
FACTS
VALUES
Facts and values: an updated perspective
“Brute” facts (House & Howe, 1999) “Bare” valuese.g. Stone is denser than cheese e.g. Pinot Gris tastes better than chardonnay
Statements that blend facts and values together
“Evaluative statements consist of fact and value claims intertwined, melded together” House and Howe,1999
Old perspective:
Updated perspective:
Recommended readingEvaluation logic Fournier, D. (1995). Establishing Evaluative Conclusions: A Distinction between
General and Working Logic. New Directions for Evaluation 68, 15-32
House E.R. (1977). The logic of evaluative argument. Centre for the study of evaluation. University of California
Values and valuing Davidson, E.J. (2005). Evaluation methodology basics. Thousand Oaks: Sage
House E. R. & Howe, K. R. (1999). Values in evaluation and social research. Thousand Oaks: Sage
Schwandt, T. A. (2002). Evaluation Practice Reconsidered. New York: Peter Lang Publishing
Argumentation Toulmin, S., Rieke, R. & Janik, A. (1979). An introduction to reasoning. New York: Macmillan Publishing
Evaluative argument Fournier, D., & Smith, N. L. (1993). Clarifying the Merits of Argument in Evaluation Practice. Evaluation and Program Planning, 16, 315-323
Schwandt, T.A. (2008). Educating for Intelligent Belief in Evaluation. American Journal of Evaluation 29,139
Evidence Donaldson, S.T., Christie, C. A. & Mark, M.M. (Eds.), (2009). What counts as credible evidence in applied research and evaluation practice. Thousand Oaks: Sage
