PoUcy Studies Journal, Vol. 23, No. 4,1995 (712-715)

Policy Analysis Methods: Two AlternativeApproaches and One Polemic

William J. Ball

Frank Fischer, Evaluating PubUc PoUcy. Chicago, IL: Nelson Hall Publishers, 1995.Emery Roe, Narrative PoUcy Analysis: Theory and Practice. Durham, NC: Duke

University Press, 1994.William C. Mitehell and Randy T. Simmons, Beyond PoUtics: Markets, Welfare, and

the Failure of Bureaucracy. Boulder, CO: Westview Press, 1994.

Each of Uiese books is similar in type and general stmcture. Each combines aprimary emphasis on methodology in policy analysis with chapters devoted toillustrative applications. The works by Fischer and Roe are quite compatible with eachother, drawing upon interpretive approaches to bring assessment of the qualitative andnormative dimensions of policy issues to Uie more traditional empirical tools of theanalyst. The book by Mitchell and Simmons takes a completely contradictoryapproach, arguing for the public choice perspective as the sole appropriatemethodological paradigm for analysis.

Evaluating Public Policy

Fischer's book culminates years of work in defining an approach to policyanalysis that is postpositivist but more than simply critical of positivist traditions.Fischer's central concem has been how to integrate the evaltiation of the normativeprinciples with the assessment of empirically testable propositions contained in apolicy statement. In the past, he has termed this a "value-oriented" epistemology forpolicy analysis (Fischer, 1980). Methods for judging the quality of the normativepositions taken by policy advocates are emphasized in Fischer's work, something thatis both its greatest strength and most serious limitation.

Fischer draws upon two inspirations for the architectonics of his approach topolicy analysis. The first is Stephen Toulmin's (1958) schematic for argumentanalysis. The second is Paul Taylor's (1961) hierarchy of justification in ethics.Surprisingly, Toulmin's schematic is relegated to only an epilogue in EvaluatingPublic Policy, while Taylor's hierarchy is used to stmcture the entire book. Taylor'shierarchy consists of four increasingly general levels of justification: verification,vindication, validation, and social choice.

In Fischer's application to policy analysis, verification means assessing apolicy's ability to satisfy its own objectives. This primarily is an assessment to beundertaken by the traditional empirical tools of the field. Fischer gives relatively briefattention to this level, illustrating wiUi the example of the Head Start program that thetechnical-analytic discourse of verification is not as uncontentious as it may seem atfirst. He does a good job of demonstrating how the selection of which objectives toverify is an issue to be considered at the next level—validation.

Validation addresses the suitability of policy objectives to the social problemat hand. Fischer defines validation rather narrowly, as a question of relevance.Qualitative meUiods, especially interpretive ones, are best suited for validating programobjectives. Thus the author briefiy illustrates how methods like case study, directobservation, and personal interviewing can uncover less obvious policy objectives,

712

Reviews/Essays: Ball

show how objectives fit within accepted social rules, and demonstrate institutionalsupport or opposition to particular criteria for evaluation. The auUior also includes ashort discussion on the phenomenological sociology that lies behind validation, and anapplied chapter on validation using Times Square redevelopment as an example.

With vindication and social choice one moves to second-order levels ofevaluation. Attention shifts from the context of the particular policy being evaluated tothe societal system as a whole. The required disciplinary background shifts from thoseof Uie policy analyst and sociologist to Uiose of Uie systems Uieorist and politicalphilosopher. Vindication asks whetho* the overall goals of a policy serve an importantfunction for existing social arrangements. Thus vindication is essentially an empiricalprocess, which Fischer illustrates with a chapter on disability policy. Social choice,Uie last level of justification, refers to Uie selection among ways of life. This is thearena of grand normative i^ilosophy, where entire ideologies must be evaluated againsteach other on their foundadonal principles.

Using Taylor's hierarchy provides a coherent structure for integratingnormative evaluation and empirical assessments. However, it is also limiting. It leadsFischer to elevate too highly in his scheme systemic normative problems raised bypolicy issues. Taylor developed the hiwarchy for use in ethical philosophy, not foridentifying the best policy argument or for an aid in making policy choices. Use of thehierarchy implies that a policy is not justified acceptably until it has proven itselfsuccessfully at the level of social choice, and that all disputes over which policy choiceneeds to be made will be resolved ultimately only after very long-term consideration atUiis very abstract level.

This limitation aside. Evaluating PubUc PoUcy is an important work in thefield. Fischer brings a well-articulated method to an underdeveloped aspect of policyanalysis. The book is also fiexible enough to be employed in a wide variety of ways.Indeed, the author has peppered the work with intriguing short boxed comments onrelated themes. He also has included bibliographies of suggested readings at the end ofeach chapter and 20 pages of references at the end of the book. Further, Fischer'scoverage of the policy history in each of his applied examples is outstanding. More sothan in his previous works. Fisher has reached an exciting middle ground betweentheoretical treatise, application manual, and textbook in Uie way he has stmctured andwritten Evaluating Public Policy.

Narrative Policy Analysis

Roe's objective is to demonstrate that theories of narrative are "exceptionallyhelpful" in analyzing policy issues and making policy recommendations. He definespolicy narratives as "... stories (scenarios and arguments) which underwrite and stabilizethe assumptions for policymaking in situations that persist with many miknowns, ahigh degree of interdependence, and littie, if any, agreement" (p. 34). Roe offers as hiskey insight that the stories used in describing and analyzing policy issues are animportant force that must be considered in policy analysis. These stories resist changein the face of contradicting data because they provide a stabilizing force indecisionmaking.

Narrative policy analysis is a four-step process. First, the analyst mustidentify the stories that support the dominant decisionmakers in the policy area. Thesecond step is to identify other stories, those that do not support the policy regime.Here Roe writes of counterstories, which contradict Uie issue's dominant narratives, andnonstories, which are narratives lacking the elements of a complete story (i.e.,

713

PoUcy Studies Joumal, 23:4

beginning, middle, end, plot, and characters). The third step of Uie process consists ofgenerating a metanarrative. The metanarrative is created by comparing Uie stories to thecounterstories and nonstories— it identifies the nature of the confiict among the differentsides of the policy issue. In the last step of the process, the analyst attempts to makeuse of the metanarrative to make the policy issue more amenable to conventional policyanalytic techniques.

Roe's articulation of the narrative policy analysis process is quite sketchy,especially with regard to the final step of moving from narrative analysis to moretraditional techniques (the most detailed exposition of his method is contained in aseven-page aiq)endix). He does maintain consistentiy that the narrative process is not acomprehensive system for policy analysis. Rather, it is a way to begin to break apartpolicy controversies ftaught with uncertainty, complexity, and polarization for furtherstudy. Yet, even with this limited objective. Roe leaves the reader wanting morespecific methodological guidance.

One of the most interesting aspects of Narrative Policy Analysis is theselection of policy issues used to illustrate methodological concepts. The author'sexpertise is in nattual resources and environmental policy; he also has done research andconsulting in Africa. Thus Roe begins his examples with an examination of budgets astexts, but these are Third-World budgets, particularly those of Kenya and Botswana.The land-use experiences of the same nations are t^ped to criticize the tragedy of thecommons as a story that continues to underwrite policy decisions in the face ofcontradictory empirical evidence. Roe's oUier examples include the Medfly controversyin Califomia, animal rights, irrigation policy in Califomia, global warming, and theretum of Native American remains by museums.

The book is structured in a slightiy unusual manner. The introductionincludes a digression to answer possible criticisms; this would have been more useful toUie reader after Roe's approach has been laid out. Similarly, the concluding chapter isdevoted entirely to a discussion of the ethics of his s^proach to policy analysis, whichwould have been more useful in (»ienting tl^ reader early MI. The intervening chaptersillustrate narrative policy analysis—each using a different policy controversy toilluminate a different aspect of Roe's approach. While the wide variety of illustrationsmake interesting reading, the middle chapters seem unintegrated with each other—perhaps because each is based on a separate published article.

Narrative PoUcy Analysis, a relatively sh(»t work, is a worthwhile read for thefresh perspective it brings to the task of policy analysis, rather than for depth oftheoretical insight or for its methodological sophistication. It suffers greatiy on theselater points in comparison to Fischer's book. Although Roe doesn't really deliver onshowing how theories of narrative are "exceptionally helpful" in analyzing policyissues. Narrative PoUcy Analysis contains several insights into the force of languageand storytelling in politics. Although perhaps not intended entirely this way by theauthor, tactics for telling more persuasive stories abound, imparting to the worksignificant value as a weapon in policy advocacy.

Beyond Politics

The most disconcerting thing about Beyond Politics is that it isn't Theauthors, both political scientists, do not recognize this fact despite their cautiousoptimism about a growing constituency for "responsible politics," consisting of "...expanding personal liberty, reducing govemment's arbitrary power, and promotingmarket efficiency ..." (p. xix). By making politics a dirty word in this book, the

714

Reviews/Essays: Ball

authors are unable to be forthright about the inherently political nature of theirideological position. Mitehell and Simmons clearly are convinced that their po'spectiveis the only one that deserves to be called rational.

Mitehell and Simmons treat us to a very readable exposition and 2spp\icaixoa ofthe public choice perspective to public policy issues and the bureaucracy. Theapplication to policy-related issues is almost exclusively theoretical, with only the rarebit of data offered to vaify the conclusions they reach. In particular, Uiey strike back atthe idea of market failtire as a justification for increased govemment involvement.Instead, they develop the idea of govemment failure—that government and politics arepoor substitutes for individual self-interested choice ami markets, even in what iK)rmallyare considered to be political decisions. Their book is, to put it mildly, one-sided in itsoutiook.

The authors begin wiUi a lucid chapter outlining the theory of market failure.Mixing micro and macro levels of analysis, they review the concepts of extemalities,public goods, imperfect competition, economic instability, distributive inequality, andtransaction costs. This chapter, like all oUiers in the book, is supplemented wiUi a veryuseful set of bibliogr^hical notes and references.

The polemical nature of Beyond PoUtics begins to emerge with the secondchapter, which offers a weakly supported, "straw man" argument about how politicalscientists and welfare economists view politics. For example, they say Uiat politicalscientists believe bureaucrats to be well-meaning civil servants working in the publicinterest Govemment is viewed as having a neutral and passive role, waiting to respondto voters and interest groups. Only a couple of contemporary political scientists arecited to support this simplistic and outdated view—their primary expert on publicadministration is Woodrow Wilson!

The rest of the book is devoted to applying the public choice perspective to awide range of political phenomena. This is done in an eminenUy readable way, even forthose with litUe background in economics. Beyond Politics does not offer anythingnew to those familiar with the public choice literature— its intent seems to be more asan introduction, or indoctrination, to the field for noneconomists. It could be used inthe classroom with great caution, given its polemical nature. However, a far superiorwork for classroom use would be The Economist's View of the World, by StevenRhoads (1985), which provides the balanced perspective on public choice, welfareeccmomics, and political science sorely missing from Beyond Politics.

***

William J. Ball is assistant professor of political science at Trenton StateCollege.

References

Hscher, F. (1980). Politics, values, and public policy: The problem of methodology. Boulder, CO:Westview Press.

Rhoads, S. (1985). The economist's view of the world: Government, markets, & public policy.Cambridge: Cambridge University Press.

Tayl(M-,P. (1961). Normative discourse. Englewood Cliffs, NJ: Prentice-Hall.Toulmin, S. (1958). The uses of argument. Cambridge: Cambridge University Press

715