John Gerring Department of Political Science
Boston University firstname.lastname@example.org
Annual Review of Political Science 20 (May/June 2017)
Draft: 30 November 2015
Estimated pages: 23 (http://www.annualreviews.org/page/authors/article-length-estimator-1)
Please do not cite without permission
Qualitative methods, broadly construed, extend back through the foggy mists of time to the very beginnings of social and political analysis (however that might be dated). Selfconscious reflection on those methods is comparatively recent. The first methodological statements of contemporary relevance grew out of the work of logicians, philosophers, and historians in the nineteenth century, most importantly J.S. Mill (1843). To be sure, these scholars were in a quest for science, understood as a unified venture. So the notion of a method that applies only to qualitative data would have made little sense to them.
At the turn of the twentieth century, a bifurcation appeared between quantitative and qualitative methods (Platt 1992). The natural sciences, along with economics, moved fairly quickly and without much fuss into the quantitative camp, while the humanities remained largely qualitative in orientation. The social sciences found themselves in the middle divided between scholars aligned with each camp, and some who embraced both. For this reason, the qual/quant distinction has assumed considerable importance in these fields, and very little importance outside these fields.
Perhaps it is not coincidental that the quest for a method to qualitative inquiry has proceeded further in the social sciences then in the humanities. And among the social sciences one might argue that political science has gone further than any other in developing the field of qualitative methods. Accordingly, this review article focuses primarily on work produced by political scientists, with an occasional glance at neighboring disciplines.
I begin by discussing the time-honored qualitative/quantitative distinction. What is qualitative data and analysis and how does it differ from quantitative data and analysis? I propose an explicit definition for qualitative and then explore the implications of that definition. The second section focuses on several areas of qualitative research that seem especially prominent and/or fecund, judging by output over the past decade. This includes (a) case-selection, (b) frameworks for qualitative inquiry, (c) rules of thumb for qualitative inquiry, and (d) multimethod research.
Qual and Quant
Although the qual/quant distinction is ubiquitous in social science today, the distinction is viewed differently by scholars in either camp. As a rule, scholars whose work is primarily quantitative tend to view social science as a unified endeavor, following similar rules and assumptions. The naturalistic ideal centers on goals such as replication, cumulation, and consensus all of which point toward a single logic of inference (Beck 2006, 2010; King, Keohane & Verba 1994).
By contrast, scholars whose work is primarily qualitative tend to view the two modes of inquiry as distinctive, perhaps even incommensurable. They are more likely to identify with the idea that knowledge of the world is embedded in theoretical, epistemological, or ontological frameworks from which we can scarcely disentangle ourselves. They may also identify with the phenomenological idea that all human endeavor, including science, is grounded in human experience. Since experiences often couched in positions of differential power and status vary, one can reasonably expect that the methods and goals of social science might also vary. The apparent embeddedness of knowledge reinforces qualitative scholars predilection toward pluralism, as it suggests that there are fundamentally and legitimately different ways of going about business (Ahmed & Sil 2012; Bennett & Elman 2006: 456-57; Goertz & Mahoney 2012; Hall 2003; Mahoney & Goertz 2006; Shapiro, Smith & Masoud 2004; Sil 2000; Yanow & Schwartz-Shea 2013).
Following the axiom that where one sits determines where one stands, let us also consider the stakes in this controversy. Over the past century quantitative work has been on the ascendant and qualitative work has been cast in a defensive posture. There are lots of qualitative
practitioners but comparatively few qualitative methodologists. Consequently, it has been difficult for researchers to explain their work in ways that those in the quantitative tradition can understand, and respect. Uncomfortable with the prospect of absorption into a quantitative template, one may surmise that many qualitative scholars have sought to emphasize the distinctiveness of what they do for strategic reasons establishing a nature preserve for an endangered species, as it were.
Whatever its intellectual and sociological sources, the question of unity or dis-unity depends upon how one chooses to define similarity and difference. Any two objects will share some characteristics and differ in others. It follows that they may be either compared or contrasted, depending upon the authors point of view. Quantitatively inclined scholars may choose to focus on similarities while qualitatively inclined scholars choose to focus on differences. Both views are correct, as far as they go. The half-empty/half-full conundrum seems difficult to overcome in this particular context.1 To put the matter in a more specific frame: all may agree with Brady & Collier (2010) that there are diverse tools (the pluralistic perspective) as well as shared standards (the monist perspective). But they do not necessarily agree on what those shared standards are or to what extent they should discipline the work of social science.
Any attempt to resolve the monism/pluralism question that begins with high-level concepts (e.g., monism and pluralism, logic of inquiry, epistemology, commensurability, naturalism, interpretivism) is probably doomed to failure. These words are loaded, and once they have been uttered the die is cast. Participants from either camp will dig in their heels.
I propose to take a ground-level approach that avoids hot-button concepts from philosophy of science and focuses instead on matters of definition. What, exactly, is qualitative data? And what, by contrast, is quantitative data? We shall then explore the repercussions of this distinction, working toward some tentative conclusions that hopefully all may agree with, even if they do not resolve all aspects of the qual/quant debate.
Definitions Since qualitative and quantitative are antonyms one cannot define one without defining the other. I begin, therefore, by listing some of the attributes commonly associated with these contrasting methods.
Qualitative work is expressed in natural language while quantitative work is expressed in numbers and in statistical models. Qualitative work employs small samples, while quantitative work is large-n. Qualitative work is often focused on the subjective feelings and understandings of those under study and, accordingly, with techniques such as ethnography and unstructured interviews. Quantitative work is often focused on seemingly objective conditions, or things held in common. Qualitative work draws on cases chosen in an opportunistic or purposive fashion while quantitative work employs systematic (random) sampling. Qualitative work is often focused on particular individuals, events, and contexts, lending itself to an idiographic style of analysis. Quantitative work is more likely to be focused on features that (in the researchers view) can be generalized across a larger population, lending itself to a nomothetic style of analysis.
Let us suppose that all of the foregoing contrasts contain some empirical truth; that is, the foregoing characteristics co-vary in the work of social scientists. And let us further suppose that they resonate with common usage of these terms, as reflected in work on the subject (e.g., Bennett & Elman 2006; Brady 2010; Caporaso 2009; Collier & Elman 2008; Glassner & Moreno 1989; Goertz & Mahoney 2012; Hammersley 1992; King, Keohane & Verba 1994; McLaughlin 1991; Levy 2007; Morgan 2012; Patton 2002; Schwartz & Jacobs 1979; Shweder 1996; Snow 1959/1993; Strauss & Corbin 1998). If so, we have usefully surveyed the field. But we have not provided anything more than a map of this rugged terrain. 1 This is nicely illustrated in recent arguments about causation (Reiss 2009).
My goal is to arrive at a minimal definition that bounds our subject in a fairly crisp fashion, that resonates with extant understandings, and that does not trespass on other well-established terms. (It would not be efficient, semantically speaking, to conflate qualitative with idiographic, ethnographic, or some other term in this family of concepts.) In addition, it would be helpful if the proffered definition accounts for (in a loosely causal sense) the various attributes commonly associated with the terms qualitative and quantitative as surveyed above.
With these goals in mind, I propose that a fundamental feature of qualitative work is its use of non-comparable observations observations that pertain to different aspects of a causal or descriptive question. As an example, one may consider the clues in a typical detective story. One clue concerns the suspects motives; another concerns his location at the time the crime was committed; a third concerns a second suspect; and so forth. Each observation, or clue, draws from a different population. This is why they cannot be arrayed in a matrix (rectangular) dataset and must be dealt with in prose (aka narrative analysis). It is also why we have difficulty counting such observations. The time-honored question of quantitative research What is the n? is impossible to answer in a definitive fashion. Likewise, styles of inference based on qualitative data operate somewhat differently than styles of inference based on quantitative data.
I therefore define quantitative observations as comparable (along whatever dimensions are relevant) and qualitative observations as non-comparable, regardless of how many there are. When qualitative observations are employed for causal analysis they may be referred to as causal-process observations (Brady 2010), though I shall continue to employ the more general (and less bulky) term, qualitative observation, which applies to both descriptive and causal inferences.
The notion of a qualitative or quantitative analysis is, accordingly, an inference that rests on one or the other sort of data. If the work is quantitative, it enlists patterns of covariation found in a matrix of observations and analyzed with a formal model (e.g., set theory/QCA, frequentist statistics, Bayesian probabilities, randomization inference) to reach a descriptive or causal inference. If the work is qualitative, the inference is based on bits and pieces of non-comparable observations that address different aspects of a problem. Traditionally, these are analyzed in an informal fashion, an issue taken up below.
Some strategies of data collection seem inherently qualitative, e.g., unstructured interviews, participant-observation (ethnography), and archival work. This is because researchers are likely to incorporate a wide variety of clues drawn from different kinds of sources and addressing different aspects of a problem. The different-ness of the evidence makes them non-comparable, and hence qualitative. Other data collection strategies such as standardized surveys are inherently quantitative, as they involve counting large numbers of observations that are comparable by assumption. Of course, they might not actually be comparable. We are speaking here of assumptions about the data generating process, not about the truth with a capital T. But we cannot avoid assumptions about the world, and these assumptions quite rightly lead researchers to adopt one or the other method of apprehending that reality.
Converting Words to Numbers No qualitative observation is immune from quantification. Interviews, pictures, ethnographic notes, and texts drawn from other sources may be coded, either through judgments exercised by coders or through mathematical algorithms (Grimmer & Stewart 2013). By coding I refer to the systematic measurement of the phenomenon at hand reducing the information at hand to a small number of dimensions, consistently defined across the units of interest. All that is required, following our definition, is that multiple observations of the same kind be produced and (voila!) quantitative observations are born. These may then be represented in the matrix format familiar to those who work with rectangular datasets.
Of course, there are often practical obstacles to quantification. Perhaps additional sources (informants, pictures, texts) are unavailable. Perhaps, if available, they are not really comparable, or they introduce problems of causal identification (e.g., heterogeneity across cases
that could pose a problem of noise or confounding). Alternatively, it may be possible to generate additional (comparable) observations but not worthwhile, e.g., because the first observation is sufficient to prove the point at issue. Sometimes, one clue is decisive. Nonetheless, in principle, if the researchers assumptions of comparability are justified, qualitative data can become quantitative data. The plural of anecdote is data.
Something is always lost in the process of reducing qualitative information to quantitative data. One must ignore the unique aspects of each qualitative observation in order render them comparable. If one wishes to generalize across a population, ignoring idiosyncratic features of the data is desirable. But if one wishes to shed light on these heterogeneous features the conversion of qualitative to quantitative data will iron out the ruggedness of the landscape obscuring variation of theoretical interest. Information loss must be reckoned with.2
Finally, and perhaps most importantly, there is an asymmetry between qual and quant. One can convert qualitative data to quantitative data but not the reverse. It is a one-way street. Once a piece of information is rendered in a matrix template whatever unique aspects may have adhered to that observation have been lost. Data reduction is possible, but not expansion. The singular of data is not anecdote, which is to say one can never recover an anecdote from a data point.
Contrasting Affinities It follows from our discussion that the utility of qualitative and quantitative data varies according to the researchers goals.
First, qualitative data is likely to be more useful insofar as a study is focused on a single case (or event), or a small number of cases (...