Embed Size (px)
Citation preview
American Political Science Review (2020) 114, 2, 356–374
doi:10.1017/S0003055419000807©AmericanPolitical ScienceAssociation 2020. This is anOpenAccess article, distributed under the terms of theCreative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, andreproduction in any medium, provided the original work is properly cited.
When the Whole Is Greater than the Sum of Its Parts:On the Conceptualization and Measurement of Populist Attitudesand Other Multidimensional ConstructsALEXANDER WUTTKE University of Mannheim
CHRISTIAN SCHIMPF University of Alberta & University of Mannheim
HARALD SCHOEN University of Mannheim
Multidimensional concepts are non-compensatory when higher values on one component cannotoffset lower values on another. Thinking of the components of a multidimensional phenomenonas non-compensatory rather than substitutable can have wide-ranging implications, both con-
ceptually andempirically.Todemonstrate this point,we focusonpopulist attitudes that featureprominentlyin contemporary debates about liberal democracy.Given similar established public opinion constructs, theconceptual value of populist attitudes hinges on its unique specification as an attitudinal syndrome,which ischaracterized by the concurrent presence of its non-compensatory concept subdimensions. Yet this conceptattribute is rarely considered in existing empirical research. We propose operationalization strategies thatseek to take the distinct properties of non-compensatorymultidimensional concepts seriously. Evidence onfive populism scales in 12 countries reveals the presence and consequences of measurement-conceptinconsistencies. Importantly, in some cases, using conceptually sound operationalization strategies upsetsprevious findings on the substantive role of populist attitudes.
INTRODUCTION
Populism is an essential social science concept, andit is also an essentially contested concept (Mudde2017). Despite ongoing disagreement about its
content,origins, andconsequences (e.g.,Aslanidis 2016;Rooduijn 2014), populism is en vogue, and the term iswidely used among public intellectuals, politicians, andacademic scholars. In the academic realm, a recentstrand of social inquiry investigates populism at theindividual level (Akkerman,Mudde, andZaslove 2014;
Hawkins, Riding, and Mudde 2012). Often referred toas the study of “populist attitudes,” this research startsfrom the assumption that populist ideas must resonatewith the public to be influential (Hawkins and RoviraKaltwasser 2018; Spruyt, Keppens, and van Droogen-broeck 2016). Therefore, scholars examine how pop-ulist discourses, styles, and strategies among politicalelites correspond with the distribution of populist ideasamongordinary citizens (e.g.,CastanhoSilva et al. 2018;Hameleers, Bos, and Vreese 2017; Hawkins, Riding,andMudde 2012; Hieda, Zenkyo, and Nishikawa 2019;Schulz et al. 2018; Spruyt, Keppens, and van Droo-genbroeck 2016). The promise of this line of research isthat understanding populism at the individual levelmayhelp understanding populism at the societal level, thuspromoting the comprehension of how the “PopulistZeitgeist” (Mudde 2004) affects health and outlook ofpluralist democracies.
In this study, we take a step back and consider someunresolved problems regarding the concept structure ofpopulismat the individual level.Responding to criticismthat populist attitudes do not provide any theoreticalimport to the established public opinion literature,populism scholars have devotedly been discussing theconceptual core of populist attitudes, thereby achievinga notable convergence about the concept’s essentialcharacteristics (Rooduijn 2019). Most scholars nowconcur that populist attitudes denote a multidimen-sional construct, composed of two or more conceptcomponents (Akkerman, Mudde, and Zaslove 2014;Castanho Silva et al. 2018; Mudde and Rovira Kalt-wasser 2013a; Oliver and Rahn 2016; Schulz et al. 2018;Stanley 2011). Although unanimity has not yet beenreached about the exact content of these components,there is widespread agreement on the idea that populist
AlexanderWuttke , PhDStudent,Department of Political Science,University of Mannheim, [email protected].
Christian Schimpf , ResearchAssociate, Department of PoliticalScience, University ofAlberta; and PhDCandidate, Graduate SchoolofEconomic andSocial Sciences,UniversityofMannheim, [email protected].
Harald Schoen, Professor, Department of Political Science, Uni-versity of Mannheim, [email protected].
We thankFlavioAzevedo, StephanQuinlan, several staffmembersof the chair for Media Psychology & Effects (Werner Wirth, Uni-versity of Zurich), and participants at the 2018 ECPR GeneralConference, GESIS research day, and GLES project meeting forthoughtful comments. We thank the three anonymous reviewers forvery constructive advice, and Bruno Castanho Silva and colleaguesfor making their data available. We thank Denis Cohen for help withthe simulation analysis. We thank Frederic Auel and AlexanderSchreiber for help with the manuscript. This work was supported bythe German Research Foundation [SCHO 1358/4-3]. ChristianSchimpf also acknowledges the support from the University ofMannheim’s Graduate School of Economic and Social Sciencesfunded by theGermanResearch Foundation, and theGESIS LeibnizInstitute which he previously worked for. Replication files areavailable at theAmerican Political ScienceReviewDataverse: https://doi.org/10.7910/DVN/KPS1KY.
Received: November 22, 2018; revised: August 11, 2019; accepted:December 2, 2019; First published online: February 4, 2020.
356
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
attitudes lie at the intersection of the concept’s sub-dimensions (Castanho Silva et al. 2018; Hameleers, Bos,and Vreese 2017, 482; Hieda, Zenkyo, and Nishikawa2019, 3;March 2017, 283;Mudde andRoviraKaltwasser2013a, 2013b, 149; Spruyt, Keppens, and van Droogen-broeck 2016, 336, but see Hameleers, Bos, and Vreese2017, 482).1 In other words, the supposed uniqueproperty of populist attitudes is the coexistence of itscomponents. Hence, what distinguishes populist atti-tudes from similar attitudinal concepts is its status as anattitudinal syndrome, which only considers citizens aspopulists if, for instance, theyaccept anti-elitist viewsandaManicheanoutlookandbelieve inunrestrictedpopularsovereignty. However, existing studies on populism atthemass level rarely transfer this crucial concept featureinto empirical practice. Because of the apparent mis-match between the theoretically derived concept speci-fication and the concept’s operational use, manypopulism studies do notmeasure what they are intendedto measure, and therefore, the reported results do notnecessarily speak about the concept under investigation.
Although we focus on the concept of populist atti-tudes, this study also highlights the general importanceof aligning ameasure’s mathematical structure with thetarget concept’s theoretical structure for valid infer-ences in social inquiry (cf. Goertz 2006, 125) and thushas implications for other non-compensatory multidi-mensional concepts beyond research on populism.Using inadequate operationalization strategies mayyield dubious results. For instance, after thoroughdiscussions on measurement-concept inconsistencies instudies on democratization (Collier and Mahon 1993;Goertz 2006; Munck and Verkuilen 2002), scholarsbegan to derive non-compensatory measures of de-mocracy, in which countries could not substitute lowlevels of electoral fairness with high levels of minorityprotection because both components constitute nec-essary elements of liberal democracies (Møller andSkaaning 2012, 135). Similar arguments play a role inother research areas, where the presence of a conceptdepends on the simultaneous presence of all conceptcomponents such as deprivation (Alkire et al. 2015),human development (Greco et al. 2019), or democraticsupport (Schedler and Sarsfield 2007).
Using populist attitudes as an illustrative case, thisstudy shows that different operationalization strategiesof multidimensional concepts may ultimately yield di-verging substantive inferences. For the studyof populismon the individual level, deriving adequate operationstrategies is particularly important as the conceptualvalue of populist attitudes hinges on the specification asa multidimensional concept with non-substitutablesubdimensions. This article thus demonstrates howsimple adjustments of empirical practices enable theblossoming literature on populist attitudes to avoidmeasures that misrepresent the object under in-vestigation. Importantly, the presented analyticalframeworkand theoperational tools for aligningconcept
and measurement also apply to other manifestations ofpopulism, such as party strategies or communicationstyles (e.g., March 2017), and to other multidimensionalconcepts beyond populism research that are constitutedby non-compensatory concept components.
To help overcome the inconsistency between theoryand research practice, this article proceeds in severalsteps. Because any attempt to derive valid measuresrequires a solid understanding of the target concept, wefirst review debates about the essence of populist atti-tudes. We show that the idea proposed by populismscholars to ensure the concept’s field utility is largelyneglected where the concept is put into empiricalpractice. Taking a broader perspective, we then arguethat current empirical research relies on a paradigm foroperationalizing latent constructs, which—despite itsprevalence in public opinion research—is not suitablefor multidimensional concepts with non-interchange-able concept components. Therefore, we draw onclassical and modern approaches to concept formationto discuss strategies for the operationalization of non-compensatory concepts that enables us to take thisdistinct characteristic of populist attitudes seriously.Using data from 12 countries and five scales of populistattitudes, we show substantial disparities between in-dividual populism scores derived from the existingmethods compared with the new approach. Moreover,we demonstrate that these methods can lead to di-verging conclusions about the relationship of populistattitudes with substantive variables of interest.
WHAT ARE “POPULIST ATTITUDES”?
Arguably, no other question surrounding populism hasachieved as much attention as the question of whatpopulism is. This inquiry can be divided into two sub-issues. First, there are the theoretical propositions thatscholars associate with populism as a concept. Amongother things, populism has been defined as mass move-ment or distinct form of mobilization (Barr 2009; Jansen2011; Kenny 2017), as a discourse (Laclau and Mouffe1985), a thin-centered ideology (Mudde 2004), a frame(Aslanidis 2016), a style (Moffitt and Tormey 2014),a strategy or organization (Weyland 2001), a set of ideas(Hawkins 2009), as a form of political representation(Caramani 2017), or a conception of democracy (Pappas2016; Ubinati 2019). Accordingly, there is no consensusabout the essence of the concept. Second, scholars mustalso define the set of attributes that constitute populism.Dependingonhowpopulism is conceivedof, the conceptis characterized by numerous combinations from a set ofat least a dozen different attributes, ranging from anti-elitism over polarization to the centralization of lead-ership and simplistic language (Rooduijn 2014, 578).Altogether, there is ongoing uncertainty about popu-lism’s conceptual core.
However, notable conceptual convergence is ap-parent in scholarship on populism at the mass level.Most studies of populist attitudes (e.g., Akkerman,Mudde, and Zaslove 2014; Mohrenberg, Huber, andFreyburg 2019; Schulz et al. 2018; Spierings andZaslove
1 In this study, we use the terms concept components, attributes, andsubdimensions interchangeably.
When the Whole Is Greater than the Sum of Its Parts
357
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
2017) follow the ideational approach (Hawkins et al.2018) and start from the definition of populism as a so-called thin-centered ideology (Mudde 2004, 543; 2007,23). Moreover, scholars mostly agree that populist atti-tudes consist of two or more essential components orsubdimensions (Akkerman, Mudde, and Zaslove 2014;Castanho Silva et al. 2018;Muller 2017; Schulz et al. 2018;Stanley 2011). Anti-elitism is usually on the list of coredimensions (e.g., Castanho Silva et al. 2018; Mudde andRovira Kaltwasser 2013a; Schulz et al. 2018; see Supple-ment1).Disagreementpersists about theexact definitionsof the remaining components (Quinlan andTinney 2019),for which scholars have suggested, for instance, supportfor popular sovereignty (Schulz et al. 2018) orManicheanoutlookonsociety(CastanhoSilvaetal.2018). Inanycase,despite disagreement about the number and de-nomination of concept components, the vast majority ofscholars perceive populist attitudes as amultidimensionalconstruct comprising anti-elitist attitudes and furtherorientations about the role of the people.
Yet this conception of populist attitudes faces ques-tions about the concept’s theoretical import. Even morethan other branches of populism research, the study ofpopulism at the individual level is embedded in a field ofinferences, where many similar concepts already exist(Geurkink et al. 2019). In the public opinion literature,scholars for long have examined anti-elitist orientationsunder the rubrics of political cynicism(Agger,Goldstein,andPearl1961),efficacy(Niemi,Craig,andMattei1991),trust (Miller 1974), or support (Easton 1975). Similarly,orientations toward popular sovereignty and homoge-neity have been examined, for instance, in the nationalidentity (e.g.,Mader et al. 2018) and ethnocentrism (e.g.,BonikowskiandDiMaggio2016) literaturesorby studieson citizens’ process preferences (Schedler and Sarsfield2007) and orientations toward representational roles(e.g., Katz 1997). This list could be continued.
Importantly, the substantive overlap with existingpublic opinion constructs undermines the conceptualvalue of populist attitudes. Applying the criteria de-veloped by Gerring (1999) to evaluate concepts in thesocial sciences, concept differentiation and field utilitysuffer severely if populist attitudes are not more than thesum of established constructs. Concept differentiationrefers to “the clarity of [concept] borders within a field ofsimilar terms” (Gerring 1999, 376). Thus, concept differ-entiation is low when populist attitudes cannot be clearlydistinguished from related constructs. Furthermore, con-ceptual overlap undermines field utility because populistattitudes may “steal referents from neighboring terms,leaving these termsas empty” (Gerring1999, 383).Hence,if populist attitudes are to be added to the list of valuablepublic opinion constructs, the concept must bring to thetable a theoretical proposition that is distinct and new.
In response to these challenges, populism scholarsuphold that the distinct conceptual characteristic ofpopulism at the mass level is the concept’s status as anattitudinal syndrome, which is defined in terms ofnecessary and sufficient conditions, more specifically bythe simultaneous presence of the concept’s constituentcomponents (Akkerman, Mudde, and Zaslove 2014,1326; Castanho Silva et al. 2018; Elchardus and Spruyt
2016, 113, 120; Spierings and Zaslove 2017, 824; Spruyt,Keppens, and van Droogenbroeck 2016, 336; for a rareexception, see Hameleers, Bos, and Vreese 2017, 482).In the words of Hawkins and Rovira Kaltwasser, the“peculiarity of the populist set of ideas lies precisely inthe combination of these elements” (Hawkins andRovira Kaltwasser 2018, 6). By implication, whenpopulist attitudes lie at the intersection of the conceptcomponents, individual populism scores cannot be highwhen anti-elitist orientations are low even when a per-son strongly supports the remaining components ofpopulism. For instance, assuming a three-dimensionalpopulism concept, understanding populist attitudes asan attitudinal syndrome suggests considering citizens aspopulists only if they exhibit anti-elitist orientations andaManichean outlook and support popular sovereignty.Using a concept specification that treats the conceptcomponents as non-compensatory, that is, as jointlynecessary for the presence of populism at the individuallevel, the concept of populist attitudes indeed wouldrepresent more than the sum of its parts.
However, even though many scholars implicitly orexplicitly specify populist attitudes as an attitudinal syn-dromewithnon-compensatoryconceptcomponentsat thetheoretical level, few studies consider this property whenapplying the construct empirically (for exceptions, seeMohrenberg, Huber, and Freyburg 2019; Vehrkamp andWratil 2017). In many cases, the concept operationaliza-tion does not respect the necessary conditions. Instead,scholars obtain individual populism scores by computingweighted or unweighted averages across the conceptdimensions. More specifically, scholars aggregate theconcept attributesmanually by computingmean values orusing data-driven approaches such as factor analysis (e.g.,Spierings and Zaslove 2017, 831; Spruyt, Keppens, andvanDroogenbroeck 2016, 340). Regardless of the level ofmethodological sophistication, however, these aggrega-tion methods have in common that higher values on oneconcept component can compensate for low values onother subdimensions. These operationalization strategiesmay, for instance, assign high populism scores to citizenswhodonotshowslightest signsofanti-elitist sentiments. Incontrast to the concept’s theoretical specification, theseapproaches might identify citizens as “populists” whostrongly oppose giving power to the people or whostrongly favor elitist rule. Hence, compensatory aggre-gation rules neglect the one theoretical proposition of theconcept that is supposed to make it distinct from similarconcepts, that is, its statusasanattitudinal syndromeat theintersection of the concept components.
In the following, we argue that this compensatoryoperationalization approach is rooted in a measurementparadigm that is often applied to latent constructs butwhich is incompatible with constructs that presupposenecessary conditions as essential concept properties.
ON THE RELATIONSHIP BETWEENCONCEPTS AND CONCEPT COMPONENTS
Conceptswith twoormore components such as populistattitudes can be understood as multilevel constructs
Alexander Wuttke, Christian Schimpf, and Harald Schoen
358
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
with the concept’s essential core at the basic level, theconcept subdimensions or concept components at thesecond, and the indicators at the third level (Goertz2006, 2020). The psychometric literature on populistattitudes is usually concerned with the indicator level,investing considerable efforts into identifying adequateitem batteries to capture the construct of interest(Akkerman, Mudde, and Zaslove 2014; Castanho Silvaet al. 2018; Hawkins, Riding, and Mudde 2012; Schulzet al. 2018; Van Hauwaert, Schimpf, and Azevedo2019). As long as the existence of second-level conceptcomponents does not complicate the operationalizationof an attitudinal construct, the public opinion literatureoften views the unobservable concept of interest as thecommon cause of the observed measures at the in-dicator level. The general idea underlying this per-spective, often associated with Bollen and Lennox(1991), is to view entities at one level as the effects ofentities at the other level. Even though causalitymay goboth ways,2 the causal approach usually perceives ob-served measures as reflections of the unobservableconcept.
The perspective of reflective causality has empiricalimplications. Because the observed indicators are seenas effects of a common cause (the latent concept), theindicators are expected to correlate with each other(Bollen and Lennox 1991, 306; Edwards and Bagozzi2000, 158). Practically, this perspective suggests the useof factor analyses to assess the extent of inter-itemcorrelation and to exclude indicators with un-satisfactorily low correlations. Methodologically, thisapproach regards differences between constructs at thesame level as random measurement error. Randommeasurement error stems from unrelated constructs
that exert influences in addition to the common latentconstruct and would cancel out by averaging across allconstructs at one level. Consequently, the causal per-spective views disparities between lower-order con-cepts as subordinate statistical entities withoutsubstantive meaning.
The causal perspective is common in the literature onpopulist attitudeswhere it is applied to the indicator levelbut—implicitly or explicitly—also to describe the re-lationship between the concept and the concept com-ponents (e.g., Castanho Silva et al. 2019;Hieda, Zenkyo,andNishikawa 2019; Schulz et al. 2018).Whatwe call theBollen approach to the operationalization of populismatthe individual level considers populist attitudes as thecommon cause of the concept components, such as anti-elitist orientations and support for popular sovereignty(Figure 1, Panel A). Because the Bollen approach viewsthe concept components as partly interchangeable, thisperspective implies geometric or linear functions (sum-mary scores, factor analyses) to aggregate conceptcomponents into composite indices (i.e., an individualscore of populist attitudes). Empirically, the Bollen ap-proach is themost commonoperationalization approachin the literature on populist attitudes (see Supplement 1for an overview).
Even though the causal paradigm serveswell formostlatent constructs, we argue that it is inept for non-compensatorymultidimensional constructs,where eachcomponent constitutes a necessary condition for thepresence of the concept. First, even though the causalperspective prescribes a correlation between the con-cept components, no theoretical argument implies sucha correlation between the subdimensions in the case ofpopulist attitudes. We do expect the components tooverlap among some individuals as these are the indi-viduals who hold populist attitudes (see Figure 1, PanelB). Yet we also expect that there are non-populistindividuals who agree with none or only some of thecomponents of populist attitudes.Hence, the concept ofpopulist attitudes as an attitudinal syndrome describesattitudinal configurations among individuals, but it isagnostic about correlations between the concept attrib-utes. Second,disregarding largedifferencesbetween the
FIGURE 1. The Causal and the Ontological Perspectives on the Relationship Between Concept andConcept Attributes
Note: Similar illustrations were shown by Castanho Silva et al. (2018) and Wong, Law, and Huang (2008).
2 Theontological approach (Figure1,Panelb)partlyoverlapswith theconcept of formative indicators. However, the fact that Bollen andLennox (1991) call them causal indicators emphasizes the temporaland causal order that is assumed to underlie the relationship betweenconcept and formative indicators. Because we do not see anti-elitismas causing populist attitudes, we deem the ontological approacha better conceptualfit than the causal approach including its formativevariant.
When the Whole Is Greater than the Sum of Its Parts
359
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
attributes merely as a measurement artifact is in-compatible with the unique property of populist atti-tudes as anattitudinal syndrome, forwhich low levels onone subdimension cannot be compensated for byhigherlevels on another one.
Altogether, these arguments suggest that the Bollenapproach does not adequately characterize the re-lationship between the concept and its components inthe case of multidimensional concepts with non-sub-stitutable concept components such as populist atti-tudes. Instead of assuming that populist attitudes causeanti-elitist orientations andotherorientations (Figure 1,PanelA), amore intuitive understanding of the conceptsuggests that populist attitudes are anti-elitist ori-entations in combination with other orientations(Figure 1, Panel B). Put differently, the relationshipbetween the concept and the concept component is nota causal one but one of essence and identity.
AGGREGATION RULES FOR NON-COMPENSATORY CONCEPTS SUCH ASPOPULIST ATTITUDES
The identity relationship between the concept and theconcept components paves the way for alternativeoperationalization strategies for non-compensatoryconcepts, in which the aggregation rules are moreclosely aligned with the concept’s theoretical proposi-tions. Having established that interchangeable sub-dimensions are incompatible with the unique conceptproperty of jointly necessary concept components(qualifier of concept structure), different conceptstructures are conceivable (see Table 1). In the fol-lowing, we contrast two prototypical concept structuresthat differ in the sharpness of the membershipboundaries, that is, the quantifier of concept structure.The dichotomous quantifier presupposes clear bound-aries andsets anall-or-nothing structure for the concept.The continuous quantifier, in contrast, presupposesgray space between the concept poles, indicating fine-grained differences between entities.3 Hence, thequantifier defines whether the concept of populistattitudes separates populists from non-populists orwhether we distinguish individuals by the degree to
which they hold populist attitudes. If populist attitudesshould be assessed with an either-or assertion or witha more-or-less assertion is not a question of right-or-wrong. Instead, the quantifier reflects the researcher’sconception of the construct.
A dichotomous concept structure of populist atti-tudes, which takes the necessary conditions among thesubdimensions into account, amounts to a Sartorianconcept structure. Even though rarely employed in thepopulist attitude literature (but see Vehrkamp andWratil 2017), we can draw on classical approaches toconcept formation for its operationalization (Sartori1970, 1984). Specifically, the necessary conditions canbe operationalized by setting thresholds among theattributes (see Supplement 2 for a comprehensive dis-cussion of operationalization strategies). For example,one may specify that respondents need to agree orstrongly agree on all anti-elitism- and popular-sover-eignty-items to be counted as populists. If participantsdisagree with one or more items, they would be con-sidered non-populists. The intuitive appeal of suchthresholds is the transparency of the classification. Thisapproach is particularly appealing if the cutoff pointshave informational value (e.g., above themean or at thescale midpoint). Yet obvious cutoff points are oftenunavailable,which leads to an increased risk of inducingarbitrary choices or misuse of analytical discretion(Wuttke 2019). More fundamentally, human evalua-tions of any given entity usually differ by degree.Therefore, based on the reasoning that continuousmeasures better resemble themental representations ofattitudinal constructs, cognitive sciences have generallydeparted dichotomous measures of attitudes (Murphy2002). Against this backdrop, we conclude that theSartorian concept structure is a reasonable approach,but practical and epistemological reasons speak againstdichotomousmeasuresas a standard choice for researchon populist attitudes.
TheGoertz concept structuremergeselementsofbothapproaches to concept formation already discussed.Building on work by Goertz (2006, 2020), this conceptstructure employs Fuzzy Logic to combine the con-ceptual rigorousness of necessary attributes with theoperational flexibility of continuous outcomes. Hence,like the Bollen approach, the Goertzian conceptstructure views individuals as differing by the degree towhich theyholdpopulist attitudes.Yetunlike theBollenconstruct, the Goertzian approach does not classifyindividuals as highonpopulism if theydisagreewithone
TABLE 1. Prototypical Concept Structures of Populist Attitudes
Quantifier of concept structure
Qualifier of concept structure Dichotomous ContinuousNon-compensatory Sartori GoertzCompensatory Residual Bollen
Note: Inpractice, quantifierandqualifierhavemore than twomanifestations.Theprototypical conceptstructuresshown inTable1contrastnosubstitutability with medium substitutability (qualifier), yet other degrees of substitutability are also conceivable. Likewise, Table 1 contraststwo prototypical quantifiers, although ordinal quantifiers are also conceivable.
3 For a similar discussion of “degreeism” concerning populism orpopulist attitudes, seeAslanidis (2016), Pappas (2016),VanHauwaertand van Kessel (2018), and van Kessel (2015).
Alexander Wuttke, Christian Schimpf, and Harald Schoen
360
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
core component (e.g., anti-elitism). Instead, the Goertzconstruct structure requires the acceptance of all es-sential concept attributes for high populism scores.Rarely employed in practice (but see Mohrenberg,Huber, and Freyburg 2019), several options are avail-able for operationalizing Goertzian constructs (seeSupplement 2). The easiest approach is to use theminimum value of the concept subdimensions.4 Theminimum represents the logical equivalent of the in-tersection in set theory.Using theminimummayappearstatistically inefficient at first as this operationalizationseems to disregard the information on all but the lowestattribute.However, as themathematical representationof theminimum function shows (see further below), thecomputation of the minimum, in fact, does consider allattributes when identifying the attribute with the lowestvalue. More importantly, the minimum is the mathe-matical equivalent of the theoretical proposition thatthe lowest attribute determines the overall concept (cf.Goertz 2006, 138). Consequently, the Goertzian con-cept structure accounts for necessary conditions amongthe concept components while distinguishing varyingdegrees of accepting populist attitudes.
Altogether, we distinguish three approaches tooperationalizing multidimensional concepts such aspopulist attitudes. These concept structures map theconstruct on different scales (quantifier), and the oper-ationalization strategies reflect different perspectives onthe substitutability of the concept components (quali-fier). The conceptual analysis demonstrated that theBollen operationalization fails to capture the uniquetheoretical proposition of populist attitudes. We pre-sented two operationalization strategies that account forthe non-compensatory relationship of the sub-dimensions, thereby taking populism seriously as anattitudinal syndrome. Among those operationalizationstrategies, the Goertz concept structure has the advan-tage of reflecting the nuances of human attitudes. Ac-cordingly, epistemological and methodological reasonssuggest a preference for the Goertz concept structurewhen operationalizing populist attitudes.
For empirical research, it is crucial whether this resultof our analysis makes a difference for substantiveconclusions. As these approaches employ differentaggregation rules for combining the concept compo-nents into individual construct scores, it is plausible toexpect that they yield different results. However, thiscannot be taken for granted. We, therefore, turn toempirical analysis to explore whether the operation-alization strategies matter for substantive findings.
DATA AND OPERATIONALIZATION
In our analysis of commonalities and disparities be-tween the empirical representations of the Sartori,Bollen, and Goertz concept structures, we focus onthree scales of populist attitudes: the populism scales byAkkerman,Mudde, andZaslove (2014),CastanhoSilvaet al. (2018), and Schulz et al. (2018). Due to limitationsof space (see Supplement 5 on scale selection), resultson additional scales of populist attitudes (CSES scale,scale by Oliver and Rahn 2016) are reported in an in-teractive Shinyweb application. Supplement 6 providesfurther practical information on using these scales toassess populist attitudes as an attitudinal syndrome.
All three scales analyzed in this study conceptualizepopulism at the individual level as multidimensionalwith three concept components and view anti-elitism asanessential elementof populist attitudes.Yet, the scalesdiffer slightly in the specification of the other twocomponents (Schulz et al. 2018: Sovereignty, Homo-geneity; Castanho Silva et al. 2018: People-centrism,Manichean Outlook; Akkerman et al. 2014: Sover-eignty, Manichean Outlook; see Supplement 4 forquestion wordings). To give an impression of themeasurement of populist attitudes, Table 2 shows theindicators of an adaption of the Schulz et al. populismscale (see Supplement 3 for differences between orig-inal scales and their adapted versions used in this study).
Importantly, although all considered scales concep-tualize populist attitudes as multidimensional, theyemploy different strategies to account for the concept’snon-compensatory qualifier. By and large, the Schulzet al. and Castanho Silva et al. scales measure eachsubdimension separately with distinct items for eachdimension. The Akkerman et al. scale, on the otherhand, often uses double-barreled items that tap intomore than one subcomponent, so that individuals onlyscore highly when agreeing with all aspects that arementioned in a survey item (e.g., “I would rather berepresented by a citizen than by a specialized politician,”which taps into sovereignty orientations and anti-elitistorientations, see Table S6-1 for an overview of the scaleitems). The latter strategy has certain disadvantagessuch as the ambiguousmeaning of low values or greaterdifferential item functioning (see Supplement 14).However, it has the advantage of at least partly ac-counting for the concept’s non-compensatory dimen-sions before operationalization already at the stage ofmeasurement. Note, however, that the individualAkkerman et al. items do not tap into all sub-components (see Supplement 6). Therefore, groupingand aggregating the items into composite scores is stillnecessary and may thus lead to varying resultsdepending on the operationalization strategy.
For the analysis in themain text, wemakeuse of threedata sources. First,we employ theCampaignPanel 2017of the German Longitudinal Election Study(Roßteutscher et al. 2018). Respondents for this multi-wave online survey were drawn from a heterogeneousonline sample, using quotas that are representative ofthe German online population. The dataset offersa large sampleof respondents (N513,563) andarich set
4 Different aggregation strategies are conceivable, depending on theassumptions regarding the components’ substitutability, see Goertz(2006, 111ff), Møller and Skaaning (2012, 122ff), and Munck (2009,48ff). The minimum is the adequate operationalization under theassumption of non-compensatory and non-interactive attributes.However, if theconcept implies some interchangeabilityor interactionof the attributes, then the researchers may opt for more flexiblesolutions (e.g., weighted arithmetic mean, multiplication, geometricmean),whichclosely resemble the structureofnecessaryconditionsbyemphasizing the attributes with the lowest value but also giving someweight to all other observed attributes (see Supplement 2).
When the Whole Is Greater than the Sum of Its Parts
361
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
of substantively relevant variables, including an adap-tation of the populism scale by Schulz et al. (2018). Weuse populism measures from survey wave 5, which wasconducted between August 17 and August 28, 2017,one month before the German federal election (seeSupplement 7 for descriptive tables of sample charac-teristics). To assess the Akkerman et al. and Silva et al.populism scales, we use another dataset from theGerman election study (Roßteutscher et al. 2019) thatwas collected usingCAPI in the fall of 2017 among2,112respondents. In addition, we use data collected byCastanho Silva et al. (2018, 2019), which was kindlymade available by the authors. The data were collectedthrough MTurk and CrowdFlower between November2016 and March 2017 in nine countries: Brazil, France,Greece, Ireland, Italy, Mexico, Spain, the UnitedKingdom, and the United States. The sample sizes varybetween 505 (MTurk, USA) and 186 (CrowdFlower,Ireland) respondents. Note that the respondentsinterviewed in the GLES CAPI survey were randomlydrawn from local population registration, whereas theGLES campaign panel and the Castanho Silva et al.datasets were both collected online using non-proba-bility samples. In Supplement 8, we discuss potentialsample biases and compare results from probability-and non-probability-based samples, which do not in-dicate systematic disparities by sampling strategy.
At the core of the Bollen approach is the (weighted)summation of the concept subdimensions into aggre-gate populism scores, where each subdimension alsorepresents the (weighted) aggregation of multipleindicators.5 Existing studies employ various computa-tional strategies to aggregate the subdimensions intoattitudinal populism scores, from simple means to ex-ploratory or confirmatory factor analyses (see
Supplement 1). In themain text,weopted touse themosttransparent and straightforward aggregation technique:the average of the equally weighted concept sub-dimensions. More specifically, we first standardized allindicators to a mean of zero and a standard deviation ofone.Asdiscussedatgreater lengthbelow, standardizationwas conducted for all operationalization approaches toensure scale invariance of the concept subdimensions.The standardized indicators were then aggregated intosubdimensions by computing the unweighted averagescore of the indicator. To attain Bollen populism scores,we computed the unweighted mean of the conceptdimensions. Hence, according to this approach, populistattitudes form a continuous variable for which highervalues on one subdimension may compensate for lowervaluesonanothersubdimensionandforwhichan increaseonone subdimension always contributes to higher overallpopulism scores. More sophisticated aggregation meth-ods yield similar substantive results (see Supplement 9).
Bollen :¼�n
i¼1Weighti 3Componenti:
At the core of the Sartori andGoertz approach is thenecessary condition that all subdimensions require highvalues to result in high populism scores. Variousoperationalization strategies are conceivable for vari-ous degrees of substitutability between concept sub-dimensions (see Supplement 2). However, the moststraightforward operationalization strategy for a non-compensatory concept with a continuous scale (Goertzconcept structure) is to use the minimum value of theconcept subdimensions. Hence, higher values on onesubdimension do not compensate for lower values onanother subdimension. Moreover, individuals with lowscores on one concept subdimension are not assignedhigh populism scores.
Goertz :¼ min Component1; . . . ;Componentnj j:Operationalizing the Sartori concept structure
involves setting thresholds for each subdimension. Ifindividuals surpass these thresholds on all sub-dimensions, they are categorized as populists.
TABLE 2. Items to Measure Populist Attitudes
Subdimensions Item 1 Item 2 Item 3
Anti-elitism Politicians talk too much and taketoo little action
The differences between thepeople and the so-called eliteare greater than within thepeople
Politicians care about whatordinary people think [R]
Homogeneouspeople
Ordinary people are of good andhonest character
Ordinary people all pull together Ordinary people share the samevalues and interests
People’ssovereignty
The people should have the finalsay on the most importantpolitical issues by voting onthem directly in referendums
The people, not the politicians,shouldmake ourmost importantpolicy decisions
The politicians in parliament needto follow the will of the people
Note: [R] indicates a reverse-coded item. Question wording from theGerman Longitudinal Election Study using amodification of the Schulzet al. populism scale, see Supplements 3 and 4 for details.
5 In compliance with standard measurement practices, it is entirelyadequate to represent the relationship between a subdimension ofpopulist attitudes and its indicators as latent and reflective, thus to usestructural equation modeling for the operationalization of thesesubdimensions. The theoretical propositions of populist attitudes onlyrequire the specification of necessary conditions with regard to therelationship between the concept subdimensions and the resultingaggregated populism score.
Alexander Wuttke, Christian Schimpf, and Harald Schoen
362
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
Otherwise, individuals are considered non-populists.Although thresholds may differ across subdimensions,in the absence of meaningful theoretical reasons fordifferentiation, we set equal thresholds for all sub-dimensions. We opted for a threshold at the 75%percentile. Thus, populists are those individuals whoembrace each subdimension of populist attitudes muchmore strongly than their fellow citizens. This thresholdis, arguably, somewhat arbitrary and we report theresults with other thresholds in Supplement 10.
Sartori :¼1 if Component1; . . . ;Componentn > Threshold
0 else:
�
To increase the transparency of methodology andresults, we provide additional information in the Sup-plementary Files.Moreover, the aforementioned Shinyweb application allows users to analyze the data un-derlying this study with an easy-to-use interface. Usersmay compute correlations between Bollen and Goertzpopulism scores with substantive variables of interestusing additional survey samples that are not reported inthe main text. In addition, users may examine thesensitivity of different Sartori thresholds when com-puting the shares of populist citizens in various pop-ulations (see http://populism.alexander-wuttke.de).
ANALYSIS
Weexamine the degree to which different strategies forthe operationalization of populist attitudes lead todifferent empirical results in three steps. The firstanalysis examines the distribution of the differentpopulism concept structures and the relationship be-tween them. Having demonstrated substantial dis-parities between populism constructs, the nextanalytical step examines variations between countriesand populism scales. In the third step, we examinewhether different concept structures yield differentconclusions about the relationship of populist attitudeswith attitudinal variables of interest, institutional trustin particular.
We start by conducting descriptive analyses of thepopulism constructs to understand how the conceptattributes give rise to composite scores of populistattitudes. For the dataset’s large sample size, we use theGerman campaign panel survey for the first analysis.Figure 2 shows univariate distributions and bivariatecorrelations of the three subdimensions of the Schulzet al. populism scale and how they relate to theGoertzian, Sartorian, andBollen operationalizations ofpopulist attitudes.
We first examine the empirical properties of popu-lism’s concept subdimensions, inspecting the univariatedistributions on the plot’s diagonal. Anti-elitist ori-entations and support for popular sovereignty arestrongly right-skewed, whereas the acceptance of ho-mogeneity is more or less normally distributed. Thesubdimensions are weakly to highly correlated with thestrongest association between anti-elitism and popular
sovereignty. Apparently, disdain for the political elitesand support for the transfer of power into the hands ofordinary people often go hand in hand, at least amongGerman respondents in the surveyedperiod. The case isdifferent for the perception of popular homogeneity.6
Many respondents in the sample hold strong anti-elitistviews but do not perceive the populace as homoge-neous. Consequently, the subdimensions of populistattitudes are statistically related, but they representdistinct political orientations.
The unbalanced correlations pattern between thesubdimensions has implications for the aggregation ofthese subdimensions into populism scores. Because theBollen populism scores follow from averaging across allconcept attributes, it is not surprising that the Bollenconcept structure is strongly correlated with the anti-elitism and the sovereignty subdimensions whichthemselves are very similar. At the same time, it has lessin common with the homogeneity subdimension ofpopulist attitudes. In contrast, the Goertz conceptstructure correlates to a similar degree with all sub-dimensions.7 Hence, even with equal weighting factors,the level to which one subdimension is reflected in theBollen populism score varies as it depends on the rel-ative closeness of that subdimension to the othercomponents of populism.
Turning to the consistency of concept and mea-surement, the Bollen operationalization does not farewell. The Bollen operationalization is vulnerable toconcept-measurement inconsistencies if individuals arelow on one subdimension but very high on theremaining subdimensions. Due to the empirical distri-bution of the homogeneity component, this hypothet-ical scenario appears rather plausible in the Germandataset. Indeed, the joint scatterplot for the Bollenscores and the homogeneity subdimension in the bot-tom left of Figure 2 provides evidence for such incon-sistencies. To some of the respondents, the Bollenoperationalization assigns relatively high populismscores despite homogeneity scores close to zero. Con-cept-measurement inconsistencies in the Bollen popu-lism scores are most prevalent for the homogeneitysubdimension, but they can be observed for all conceptattributes. If we take the argument seriously thatindividualsmust accept all concept attributes to achievehigh populism scores, then observing individuals withlow values on one subdimension but fairly high popu-lism scores is evidence that the Bollen approach fails tooperationalize populism’s essential theoreticalpropositions.
6 It is conceivable that the peculiar role of the homogeneity dimen-sionsmight result fromdiscrepanciesbetween theoriginal Schulz et al.scales and the adapted version of the instrument that is employed inthe GLES Campaign Panel. Yet, other studies (see Hieda, Zenkyo,and Nishikawa 2019) and the Castanho Silva et al. replication datasetin which these modifications to the Schulz et al. scale were not un-dertaken show similar patterns for the homogeneity dimension (seeSupplement 3).7 This result is due to the standardizationat the indicator level thatwasconducted before aggregation. If no standardization is applied, thenthe subdimensions with the highest item difficulty will correlate moststrongly with the Goertzian populism score.
When the Whole Is Greater than the Sum of Its Parts
363
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
This kind of concept-measurement inconsistencydoes not occur for the Sartorian and Goertzian conceptstructures. Examining the Sartori composite scores, thehistograms in the plot’s lowest row show that Sartoripopulists hold anti-elitist views and support popularsovereignty and view the populace as a homogeneousunit. About five percent of the sample surpasses thethreshold on all subdimensions andwere categorized aspopulists. Consequently, populists exist at the masslevel and proper operationalization enables measuringthe share of populists within a society. If populist atti-tudes are conceivedof as continuous,we canemploy theGoertz concept structure. Inspecting the joint distri-bution of the Goertz populism scores and the conceptcomponents, one characteristic of the plot signals thedesired properties of the Goertz concept structure:There is a clear-cut bisecting line in all these
scatterplots,which is the graphical equivalent of the rulethat populism scores must not be high if an individual islow on one subdimension. Hence, the Sartori andGoertz operationalization strategies ensure that highpopulism scores are assigned only to individuals whoexhibit high values on all concept components.
Having established that not all concept structuresrespect the concept’s theoretical propositions of pop-ulism at the mass level, we assess the empirical com-monalities and differences between these conceptstructures directly. The boxplots in the right-handcolumn report the distribution of the Goertzian andBollen scores among populists and non-populistsaccording to the Sartori operationalization. AmongSartori populists, the average Bollen populism score islower than the average Goertz populism score. Moreimportantly, among non-populist respondents, there
FIGURE 2. Distributions of and Correlations Between Concept Structures of Populist Attitudes andConcept Attributes (Germany, Schulz et al. Populism Scale)
Note: The scatterplots on the left show the joint distributions of the variables, which are labeled at the top of each column and on the right ofeach row. The variable at the column top is plotted on the scatterplot’s x-axis. Histograms show the distribution of the dichotomous Sartorimeasure. The plot’s diagonal row shows the univariate distribution of the variables labeled at the column’s top. The plot’s upper right panelsshowPearson’s R correlation coefficients between continuous variables and boxplots for the Sartori measure. Data from the German panelsurvey. Plots were created using the ggally R-package.
Alexander Wuttke, Christian Schimpf, and Harald Schoen
364
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
are more individuals with high Bollen populism scoresthan individuals with high Goertz populism scores. Inother words, the Bollen operationalization assigns highpopulism scores to several respondents whom theSartori approach classifies as non-populists.
Finally, we compare the Bollen and Goertz conceptstructures. TheGerman dataset shows a correlation of r5 0.89 between the measures, which is usually con-sidered a strong association. We discuss the strength ofthis relationship at greater length below, but here itsuffices to say that the correlation is notably differentfrom 1. Hence, comparing the Bollen and Goertzpopulism scores leads to the first important conclusion:the operationalization strategy that most current pop-ulism studies employ does not always match the pop-ulism scores of theGoertz concept structure, which alsoentails a continuous scale but which strictly adheres tothe concept’s theoretical propositions.
What is more, the discrepancies between the meas-ures have systematic origins. As the scatterplot shows,the discrepancies mainly occur because the Bollenoperationalization assigns high values to individualswith low Goertz scores. As we saw before, these casesrefer to respondents for which the Bollen operation-alization violates the concept’s theoretical propositionnot to assign high values if one concept subdimension islow. Consequently, when the Bollen populism scoresexceed the Goertz scores, then only the latter capturethe distinctive position of populist attitudes at the in-tersection of its concept components. In contrast, inthese cases, the Bollen scores tap into a mix of attitu-dinal concepts that are related to populism at the masslevel but donot reflect the concept itself. Put differently,in case of a mismatch, the Bollen operationalizationdoes not measure what it intends to measure, and theGoertz scores should be preferred.
So far, these results only speak to the distribution ofpopulism scores as derived from one populism scale inone country. To examine the generalizability of thesefindings, the following analyses broaden the perspectiveby turning to the Castanho Silva scale in the UnitedStates, the largest sample in the Castanho Silva et al.replication dataset. Figure 3 shows that the anti-elitistsubdimension of the Castanho Silva et al. scale is right-skewed, whereas the distribution of Manichean outlookorientation is left-skewed. Notably, these concept com-ponents of the Castanho Silva et al. scale are barelycorrelated.That is, knowingaperson’s stanceonpoliticalelites does not help predict her perception of good andevil in society.Thefact that the subdimensionsofpopulistattitudes are almost orthogonal in the US survey isnoteworthy from a methodological perspective. TheBollen approach conceptualizes a person’s latent pop-ulist inclination as the common cause of the conceptsubdimensions, which implies that the subdimensionsshould be correlated. Therefore, a lack of correlationsamong them raises doubts about the psychometricquality of the measurement. In contrast, from the ri-valling perspective whether an attitude is an essentialcomponent of populism at the individual level is not anempirical but an ontological question. Consequently, theapproach that underlies the Sartori and Goertz
operationalizations is fully compatible with low cova-riances. From this perspective, the implication of weaklycorrelated concept components merely is that few indi-viduals hold populist attitudes because in most cases thebuilding blocks of this concept are not jointly present.8
The magnitude of the covariance between the sub-dimensions affects the composite scores derived fromcompensatoryor non-compensatory operationalizationstrategies.9 Lower covariances imply a larger share ofindividuals who exhibit low values on one conceptsubdimension but high values on the remaining sub-dimensions. Because this configuration of attitudesleads to differences between the Goertz and Bollenpopulism scores, low component covariances tend to goalong with higher disparities between composite pop-ulism scores (for simulations of this interdependency,see Supplement 11). Against this backdrop, it is littlewonder that the correlation between the Bollen andGoertz scores amounts to r5 0.64 in theUSdatasetwiththe Castanho Silva et al. scale and is thus considerablylower than in the German dataset with the Schulz et al.scale (r 5 0.89), considering that the covariance be-tween components is much higher for the Schulz et al.scale. Hence, the weaker the correlation between thesubdimensions, the larger the concept-measurementinconsistency of the Bollen operationalization.
This interdependency between the concept compo-nents and the composite scores has substantive ram-ifications for the extent of resulting disparities betweenoperationalization strategies in research practice.Keeping in mind that not all populism scales specifypopulist attitudes in the sameway, composite scores aremost sensitive to operationalization strategies if scalesconceptualize populist attitudes as constituted by con-ceptually remote attitudinal orientations. As scaleproperties also vary between samples, another impli-cation is that composite scores of non-compensatoryconcepts are most sensitive to operationalizationstrategies in populations with lower covariances be-tween the concept components.
That the extent of measurement concept-incon-sistencies varies across scales and countries is furtheremphasized by the internal structure of the Akkermanet al. populism scale, as observed in the Germanprobability-based sample (Figure 4). Presumably partlydue to the scale’s particular measurement strategy, thediscrepancies between theBollen andGoertz scores arethe lowestof all the scales (r50.91).As the scalealreadyconsiders, inpart, thenon-substitutabilityof the conceptsubdimensions at the measurement stage, it makes lessof adifferencewhetherwechooseanoperationalizationstrategy that is adequate for non-compensatory con-cepts. Note, however, that operationalizing the
8 Indeed, using 75th percentiles on theUS dataset, theGoertz conceptstructure yields far less populists on theCastanhoSilva et al. scalewithweaklycorrelatedsubdimensions (0.6%) thanon theSchulzet al. scale(5.1%) with more strongly correlated subdimensions.9 Due to limitations of space, in the discussion here and below, wefocus on the Bollen measure (for its frequent usage in the literature)and the Goertz measure (for its practical and epistemological quali-ties) but disregard the Sartori measure.
When the Whole Is Greater than the Sum of Its Parts
365
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
Akkerman et al. scale with either the Goertz or Bollenapproach still does not yield identical results as theircorrelation coefficient is notably different from one.
Tobroadentheviewon thevariationofdiscrepanciesbetween operationalization strategies in differentsurvey samples and populism scales, the next step ofthe analysis uses the Castanho Silva et al. replicationdataset with survey data from nine countries. Figure 5shows correlations (Pearson’s R) between the Goertzand Bollen concept structures for theAkkerman et al.,Schulz et al., and Castanho Silva et al. populism scales.The figure provides two significant insights.
First, the figure underscores the modest but notablevariation in the similarity of the populism constructsdepending on which scale is used and where the surveywas conducted.TheGoertz andBollen scoresof populistattitudes usually correlate at r5 0.8whenmeasuredwiththe Schulz et al. scale. TheAkkerman et al. scale exhibits
slightly higher correlations inmost cases. In comparison,the correlations of the composite scores of the CastanhoSilva et al. scale lies on a lower level, between r5 0.39 inFrance and r 5 0.63 in the United States. Figure 5 alsoshows some country differences. For instance, the pop-ulism constructs are more strongly correlated in theItalian sample than in other samples. Consequently, forfuture research on populist attitudes, it is important tonote that theempirical implicationsof operationalizationdecisions differ between scales and samples.
Second, the results show that the discrepancies be-tween theGoertzian and theBollen populism scores areoften substantial. None of the discovered correlationscomes close to a perfect correlation of r5 1. In fact, allcorrelations in Figure 5 are smaller than r 5 0.9, andmany depicted correlation coefficients hover between r5 0.5 and r 5 0.8. Correlations of this magnitude areusually conceived of asmoderate to strong associations.
FIGURE 3. Distributions of and Correlations Between Concept Structures of Populist Attitudes andConcept Attributes (United States, Castanho Silva et al. Populism Scale)
Note:Thescatterplotson the left showthe jointdistributionsof thevariableswhichare labeledat the topofeachcolumnandon the rightofeachrow. The variable at the column top is plotted on the scatterplot’s x-axis. Histograms are used to show the distribution of the dichotomousSartori measure. The plot’s diagonal row shows the univariate distribution of the variables labeled at the column’s top. The plot’s upper rightpanels show Pearson’s R correlation coefficients between continuous variables and boxplots for the Sartori measure.
Alexander Wuttke, Christian Schimpf, and Harald Schoen
366
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
However, when evaluating the correlation coefficientspresented here, we have to keep in mind that thesecorrelations do not pertain to different constructs.Rather, both of the correlated scores allegedly capturethe same concept. Specifically, both populism scoreswere derived from the same data-generating process,and they only differ in the aggregation rule that wasapplied. Against this backdrop, even a correlation of,for example, r 5 0.8 between the Goertz and Bollenscores is notable. A correlation of 0.8 means thatknowing one variable only allows us to predict twothirds (64%)of thevarianceof theotherpopulismscore.Using a real-world analogy, the statistical associationbetween populism constructs derived from the sameindicators is about as strong as the correlation betweenarm-span and height (Reeves, Varakamin, and Henry1996). In other words, the Bollen and the Goertz
populism scores can be considered similar for manyindividuals, particularlywhen theAkkerman et al. scaleis used, but they are far from identical when the Schulzet al. scale is used, and the scores differ even morenoticeably for the Castanho Silva et al. scale.
So far, the analysis has established that the Bollen andGoertz constructs do not always derive identical populismscores, and when they diverge, the Bollen operationali-zation does not do justice to the essence of populism as anattitudinal syndrome with non-compensatory sub-dimensions. However, given the moderate to strong cor-relations, it ispossible thatbothconceptstructuresyieldthesame substantive results when analyzing the determinantsandconsequencesof populist attitudes. If thiswas the case,the distinction between the concept structures would pri-marily be of theoretical value but less critical for appliedempirical work, apart from the distribution of populist
FIGURE 4. Distributions of and Correlations Between Concept Structures of Populist Attitudes andConcept Attributes (Germany, Akkerman et al. Populism Scale)
Note:Thescatterplotson the left showthe jointdistributionsof thevariableswhichare labeledat the topofeachcolumnandon the rightofeachrow. The variable at the column top is plotted on the scatterplot’s x-axis. Histograms are used to show the distribution of the dichotomousSartori measure. The plot’s diagonal row shows the univariate distribution of the variables labeled at the column’s top. The plot’s upper rightpanels showPearson’sRcorrelation coefficients betweencontinuous variables and boxplots for theSartorimeasure.Data from theGermancross-sectional election survey.
When the Whole Is Greater than the Sum of Its Parts
367
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
attitudes. Hence, we now turn to explore whether thechoice between concept structures makes a difference inexplaining other substantive variables.
Although it is impossible to address all relevant ques-tionstackledinpriorresearch,weselectedsomethat, inourview, are of interest to empirical scholars in populism. Toreduce analytical discretion and to enhance transparency,we opted for the simplest model possible: bivariate cor-relations.Due to limitations of space, we only show resultson theCastanho Silva et al. scale in themain text using theauthors’ cross-national dataset. Yet, the Shiny web ap-plication enables readers to investigate statistical associa-tionsbetweencompositescoresofpopulistattitudesandanextensive list of correlates using different datasets andscales. By and large and for most scales and samples, theanalyses reveal weaker associations with Bollen populismscores compared with the Goertz construct of populistattitudes.Although thesedisparitiesoftendonotaffect thesubstantive conclusions, in a few extreme instances, thecorrelations even point in different directions. The Cas-tanhoSilvaetal. scaleseemsmostsensitivetodifferences inoperationalization strategies. Yet instances in which the
substantive conclusion suggested by the Bollen constructdoes not squarewith findingswithGoertz populism scorescan be observed for all scales.
In the following analysis, we employ the CastanhoSilva et al. replication dataset. The dataset only containstwo potential correlates (conspiratorial thinking, in-stitutional trust) but covers nine countries, thus dem-onstrating the robustness of findings in multiplesamples. As reported in Supplement 12, correlations ofGoertz and Bollen populism scores with conspiratorialthinking differ notably in size when they have the samesign, but often the correlations even point in oppositedirections. In the following, we report findings for thesecond available variable, institutional trust. In-stitutional trust is measured by a summary score of trustin political parties, government, and parliament.10
FIGURE 5. Correlations Between Goertz and Bollen Concept Structures of Populist Attitudes
10 In Supplement 13, we examine the internal consistency of thesummary score on institutional trust. Cronbach’s alpha is around 0.85inmost samples but around 0.65 in theUnited States and in theUnitedKingdom, suggesting that the structure of perceptions of these insti-tutions differ between countries.
Alexander Wuttke, Christian Schimpf, and Harald Schoen
368
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
Institutional trust is a particularly interesting casebecause for this variable we might for theoretical rea-sons expect differences in the results for the Bollen andtheGoertz concept structures. This expectation derivesfrom the following reasoning. It is well established thatstronger anti-elitist sentiments go hand in hand withlower levels of trust in political and societal institutions(Erber and Lau 1990). After all, it is political elites thatusually run these institutions. Beyond anti-elitism,however, it is more difficult to reason a substantive linkbetween institutional trust and populist attitudes. It isnot self-evident why other attributes of populistattributes (but see Urbinati 2019), such as belief inahomogeneouscitizenry,might goalongwithaperson’slevels of institutional trust. Most importantly, it is notself-evident why the intersection of these attitudesshould be associated with trust in institutions. As theBollen construct reflects the average of all sub-dimensions, the respective populism scores react tochanges on one subdimension regardless of the values
on the other subdimensions. Bollen scores will thusreflect the correlation of institutional trust and anti-elitism in any case. The Goertz construct, by contrast,considers all concept components in combination andthus does not reflect changes of the subdimensionsabove the minimum. If anti-elitism happens to be notthe subdimensionwith theminimum score, variation onit is irrelevant for Goertz populism scores. As thesubdimension with the strongest relationship to in-stitutional trust does not feed into the Goertz populismscores for all respondents, it is straightforward to expectthat the Bollen scores exhibit stronger correlations withinstitutional trust than the Goertz scores do.
Figure 6 displays bivariate correlations between in-stitutional trust and theBollen andGoertz constructs ofpopulist attitudes as well as with the concept’s sub-dimensions. As expected, there is a consistent negativeassociation between anti-elitist orientations and trust inpolitical institutions. Institutional trust is not associatedwith a Manichean outlook and not consistently
FIGURE 6. Bivariate Correlations with Institutional Trust (Castanho Silva et al. Populism Scale)
Note: In all samples, the difference between the correlation sizes of the Bollen and the Goertz concept structures is statistically significant(computed with the CoCor R-package).
When the Whole Is Greater than the Sum of Its Parts
369
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
associated with people-centrist attitudes. Hence, onesubdimension of populist attitudes exhibits the expec-ted link with the concept of interest, whereas the othersubdimensions do not.
Figure 6 also shows a statistically significant corre-lation between populist attitudes and institutional trustfor the Bollen populism scores. Considering themathematical formula underlying the Bollen concept,it follows that the aggregated populism constructinherits partial correlations of the concept sub-dimensions. Using the Bollen operationalization blursthe distinction between the subdimensions of populistattitudes and the concept of populist attitudes itself asthe conjunction of its concept components. Replicatingprevious findings in the public opinion literature onpolitical cynicism (e.g., Erber and Lau 1990), it is notsurprising to observe an attitudinal linkage betweena person’s alienation with political actors (anti-elitism)and the disdain for the organizations they represent(institutional trust). However, if the correlation of trustwith the Bollen populism score is merely a remnant ofthe association with anti-elitism, then the reportedcorrelation would represent the replication of oldfindings with a new name. The Bollen construct thuscannot distinguish whether a finding reflects the uniqueproperties of populism or only that of one sub-dimension. Put differently, theBollen concept structureleaves unclear whether a finding obtained with theconstruct is driven by the concept of interest, populistattitudes.
The Goertz concept structure captures the correla-tion between institutional trust and populist attitudesunderstood as an attitudinal syndrome rather than theshared variance of its subdimensions. Using this con-struct, different conclusions about populism’s re-lationshipwith institutional trust emerge. In eight out ofnine countries, Figure 6 shows no meaningful correla-tionbetween institutional trust and theGoertzpopulismconstruct. Hence, the correlation observed when usingthe Bollen construct disappears if populism is ade-quately operationalized. In substantive terms, as manynon-populists are equally critical of elites the analysisdemonstrates that populist attitudes are not related toinstitutional trust.
The finding that different operationalization strate-gies of populist attitudes may yield inconsistent resultswhen applied in correlative analyses is the keystone ina longer series of assessments we have conducted in thisarticle. The preceding analyses demonstrated thatcomposite scores of populist attitudes can differ toa smaller or larger degree, depending on the propertiesof scales and samples. These differences were shown tooccur in cases when theBollen concept structure fails tocapture the concept’s core proposition but instead tapsinto related attitudinal constructs that are representedin populism’s subdimensions. While not necessarily thecase, sometimes these concept-measurement incon-sistencies of the Bollen approach can have con-sequences for substantive research because they canlead to wrong judgments about the nature and corre-lates of populism at the mass level. Consequently,scholars are well-advised to employ the Goertz or
Sartori concept structures that are capable of reflectingpopulism as an attitudinal syndrome to capture theessence of populist attitudes in their empirical analyses.
CONCLUSION
The notion of a “populist Zeitgeist” (Mudde 2004) hasgiven rise to a quickly growing research field on theprevalence, causes, and consequences of populist atti-tudes (e.g., Akkerman, Mudde, and Zaslove 2014;Tsatsanis, Andreadis, and Teperoglou 2018; VanHauwaert, Schimpf, and Azevedo 2018). What makespopulist attitudes valuable as a concept is the claim thatpopulismat themass level representsmore than the sumof its parts. In its distinctive position at the intersectionof the concept components, populist attitudes are notjust another variant of concepts well-known to scholarsof public opinion. This concept structure has to be takeninto accountwhen operationalizing populist attitudes inempirical research. Research thus far, however, oftenneglected the key characteristic of the concept, namely,that populist attitudes are an attitudinal syndrome withnon-substitutable subdimensions. As a consequence,themathematical structureofmeasuresof populismandthe concept’s theoretical structure fall apart (cf. Goertz2006, 125).By implication, populismconstructs oftendonot reflect what they are intended to measure. Pre-vailingpractices thusput at risk the concept’sfieldutilityand the differentiation from established concepts inpublic opinion research. In effect, critics may askwhether research on populist attitudes is just oldwine innew bottles and may call for abandoning research onpopulist attitudes (cf. Geurkink et al. 2019). We thinkthis conclusion would be premature because populistattitudes are an original concept and empirical researchon it may provide valuable insights, once the distinctivenature of the concept is taken seriously.
Toovercome thecurrent stateof affairs and toprotectresearch on populist attitudes against unjustifiedaccusations, we proposed operationalization techni-ques that are capable of preserving the concept’s uniqueproperties in empirical applications. We identified twosuitable approaches. Both the Goertz and the Sartoriapproaches properly account for the necessary con-ditions of populist attitudes and are thus superior to theBollen approach. They differ, however, in theassumptions about the nature of attitudes. While theSartori approach builds on a dichotomous conception,the Goertz approach presupposes a continuous con-ception that is more adequate in attitudinal researchand, therefore, the preferred strategy to operationalizepopulist attitudes. Considering that both approachesare not by necessity computationally more demandingthan the widely used Bollen approach, this study sug-gests that for proper empirical analyses it is at least asimportant to align conceptual reasoning with compu-tational practices as to demonstrate technical sophis-tication (e.g., Sartori 1970).
As applications of multidimensional concepts blos-som inmany subfields of political science, the discussionof concept structures in this articlemay help to carefully
Alexander Wuttke, Christian Schimpf, and Harald Schoen
370
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
consider the theoretical properties of concepts withmultiple concept componentswhencreatingorapplyingindicators in empirical research (see Supplement 2 forspecific advice). This applies to populism research be-yond the individual level, for example, populism ofparties (March 2017; Mudde and Rovira Kaltwasser2013a) and otherfields such as research on social capital(e.g., Putnam 2000). Other complex and currently de-bated concepts such as support for democracy (e.g.,Schedler and Sarsfield 2007), ideological orientations(Converse 2006), political sophistication (e.g., Luskin1987), public attitudes toward globalization (e.g.,Mader, Steiner, and Schoen 2019), and political ex-tremism (e.g., Jungkunz 2019) are among the topics forwhich this suggestion may prove fruitful. In these andother fields of research, it seems worthwhile to considerwhether central concepts involve non-compensatorysubdimensions and whether prevailing operationali-zation strategies sufficiently account for these conceptproperties.
The evidence demonstrated that the Bollenapproach’s vulnerability to concept-measurementinconsistencies can make a considerable difference forsubstantive conclusions about the prevalence, causes,correlates, and consequences of populist attitudes. Thisfinding raises issues concerning the validity of results inprior research.Thegoodnews is that theevidence in thispaper demonstrated that the aggregation rule does notalwaysmake a substantial difference. In some cases, theBollen and Goertzian concept structures are highlycorrelated and may thus lead to similar substantiveconclusions; in other cases, this does not apply (seeShiny web application: http://populism.alexander-wuttke.de). Whether measurement-inconsistenciesoccur and induce biases in correlative analyses variesbetween populations and populism scales. It is not theaimof this study to evaluate existing populism scales butto provide guidance for their proper operationalization.Nonetheless, the general pattern is noteworthy: non-compensatory operationalization strategies are mostconsequential for scales with low covariances betweenthe subdimensions. In contrast, scales are less prone toconcept-measurement inconsistencies when they havefew subdimensions or already consider the non-com-pensatory properties at the stage of measurement eventhough the latter approach may result in a whole set ofnew challenges. In addition, measurement-incon-sistencies of the Bollen concept structure are morelikely to bias results of correlative analyses if onesubdimension of populist attitudes drives the sharedvariance with a variable of interest. The relevance ofconcept-measurement misfit in prior research for sub-stantive findings thus depends on empirical dis-tributions. Our analysis may thus be read as a call forsystematic testing of the validity of prior results. This, inturn, may make research on populist attitudes evenmore vibrant and foster its comparative outlook.
Taking populist attitudes seriously as a non-com-pensatory concept has implications for their role inpolitics.Given thenature of populist attitudes as lying atthe intersection of different components, they resemblepolitical ideologies that are at the core of political belief
systems (Converse 2006; Gerring 1997; Kinder andKalmoe 2017). As holding a coherent ideology is quitechallenging, it is thus little wonder that we foundpopulist attitudes to be not widely held by citizens.However, those citizens who hold populist attitudes ina strict sense subscribe to not only one but also severalideas that have a strained relationshipwith pluralist andrepresentative forms government (Caramani 2017;Muller 2017;Urbinati 2019). These appear to be peopleon which we cannot count when it comes to defendingliberal democracy against its foes. What is more,strongly interconnectednodesof political belief systemssuch as ideologies are more resistant to persuasiveinfluences, more easily accessible, and more conse-quential in affecting other attitudes and behaviors(Petty and Krosnick 1995). It is thus implausible toexpect citizens who subscribe to populism sensu strictoto be very responsive to attempts at making themmoresupportive of core ideas of representative democracy.In a sense, a more adequate measure of populist atti-tudes may thus paint a paler and brighter picture of thesocietal foundations of democracy: populist attitudesmay be less widespread but more resistant to changethan suggested by previous research.
As this paper sought to investigate the relationshipamong conceptual, methodological, and substantiveissues, we would like to highlight the implications ofa subtle methodological decision. Before aggregatingthem into populism scores, all items were z-standard-ized. This decision is consequential because standard-ization changes the substantive meaning of themeasures from absolute to relative quantities. For in-stance, after standardization, the Sartori conceptstructure classifies those individuals as populists whoaccept all subdimensions of populism much morestrongly, compared with the average respondent. Usingrelative measures raises two conceptual and empiricalissues. Conceptually, some approaches to conceptformation might hold that a concept’s boundary con-ditions must always be defined in absolute instead ofrelative terms (Sartori 1970). Empirically, relativistmeasures hinder the comparison of populism acrosstime and space. Considering these arguments againststandardization, it should be noted that standardizationis not mandatory. For the Sartori concept, setting ab-solute thresholds is feasible based on the indicators’substantive meaning and the researcher’s specificationof populist attitudes. Likewise, researchers can use theminimum of unstandardized concept subdimensions toset the Goertzian construct. Unstandardizedapproaches, however, rest on the assumption of com-parable measurement scales across dimensions. Thisassumption is not necessarily met, however, becauseresearchers may have selected items with different itemdifficulties which affect the distributions. If differencesin the distributions of subdimensions solely result fromresearcher choices during scale development anddonotreflect substantive differences between the respondentstoward the subcomponent’s essence, then the aggre-gation of unstandardized indicators is problematic.Such scoring issues are well known to qualitativeresearchers under the rubric of calibration (Goertz
When the Whole Is Greater than the Sum of Its Parts
371
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
2020) and have also gained attention in quantitativestudies (Kolen andBrennan 2014). In particular, recentadvance in Item Response Theory (DeMars 2016; Liuand Chalmers 2018) appears to show a promising wayforward to operationalize non-compensatory multi-di-mensional constructs without neglecting issues of scaleincomparability. To conclude, standardization lowersthe burden of assumptions about the data-generatingprocess, but it has implications for the meaning of thederivedmeasures. Researchers interested in comparingpopulist attitudes over time or in quantifying absolutelevels might prefer absolutemeasures, but in turn, mustpay close attention to the substantive meaning of theindicators and the scales that are used.
Altogether, populism has solidified its important rolein contemporary politics. Given its substantive signifi-cance, the concept deserves careful analysis and ade-quate measurement in empirical applications to avoidbiased estimates and invalid conclusions. Our analysisdemonstrated that populist attitudes should be treatedas an attitudinal syndrome that ismore than the sum (oraverage) of its subdimensions. Otherwise, it would bedifficult to justify populist attitudes as a specific conceptin public opinion research. Obtaining valid compositescores for this concept requires operationalizationstrategies that differ from those widely used in previousresearch. By providing these conceptual and method-ological suggestions, this article may prove helpful inpaving the way for fruitful empirical research on animportant political phenomenon of this era.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, pleasevisit https://doi.org/10.1017/S0003055419000807.
Touse the study’s ShinyWebApplication, visit http://populism.alexander-wuttke.de/.
Replication materials can be found on Dataverse at:https://doi.org/10.7910/DVN/KPS1KY.
REFERENCES
Agger, Robert E., Marshall N. Goldstein, and Stanley A. Pearl. 1961.“Political Cynicism: Measurement and Meaning.” The Journal ofPolitics 23 (3): 477–506.
Akkerman, Agnes, Cas Mudde, and Andrej Zaslove. 2014. “HowPopulist Are the People?Measuring Populist Attitudes in Voters.”Comparative Political Studies 47 (9): 1324–53.
Alkire,Sabina, JamesE.Foster, SumanSeth,MariaE.Santos, JoseM.Roche, and Paola Ballon. 2015. Multidimensional Poverty Mea-surement and Analysis. Oxford: Oxford University Press.
Aslanidis, Paris. 2016. “Is Populism an Ideology? A Refutation anda New Perspective.” Political Studies 64 (1S): 88–104.
Barr, Robert R. 2009. “Populists, Outsiders and Anti-EstablishmentPolitics.” Party Politics 15 (1): 29–48.
Bollen, Kenneth, and Richard Lennox. 1991. “ConventionalWisdomon Measurement: A Structural Equation Perspective.” Psycho-logical Bulletin 110 (2): 305–14.
Bonikowski, Bart, and Paul DiMaggio. 2016. “Varieties of AmericanPopular Nationalism.”American Sociological Review 81 (5): 949–80.
Caramani, Daniele. 2017. “Will vs. Reason: The Populist and Tech-nocraticFormsofPoliticalRepresentationandTheirCritique toPartyGovernment.” American Political Science Review 111 (1): 54–67.
Castanho Silva, Bruno, Ioannis Andreadis, Eva Anduiza, NebojsaBlanusa, Yazmin M. Corti, Gisela Delfino, Guillem Rico, SaskiaRuth, Bram Spruyt, Marco R. Steenbergen, and Levente Littvay.2018. “Public Opinion Surveys: A New Scale.” In The IdeationalApproach toPopulism:Concept, Theory, andAnalysis, eds.KirkA.Hawkins, Ryan Carlin, Levente Littvay, and Cristobal RoviraKaltwasser. London: Routledge, 150–78.
Castanho Silva, Bruno, Sebastian Jungkunz, Marc Helbling, andLevente Littvay. 2019. “An Empirical Investigation of SevenPopulist Attitudes Scales.” Political Research Quarterly: 1–16.Published online 12 March 2019.
Collier, David, and James E. Mahon. 1993. “Conceptual ‘Stretching’Revisited: Adapting Categories in Comparative Analysis.”American Political Science Review 87 (4): 845–55.
Converse, Philip E. 2006. “The Nature of Belief Systems in MassPublics (1964).” Critical Review 18 (1–3): 1–74.
DeMars, Christine E. 2016. “Partially Compensatory Multidimen-sional Item Response Theory Models: Two Alternate ModelForms.” Educational and Psychological Measurement 76 (2):231–57.
Easton, David. 1975. “A Re-assessment of the Concept of PoliticalSupport.” British Journal of Political Science 5 (4): 435–57.
Edwards, Jeffrey R., and Richard P. Bagozzi. 2000. “On the NatureandDirectionofRelationships betweenConstructs andMeasures.”Psychological Methods 5 (2): 155–74.
Elchardus, Mark, and Bram Spruyt. 2016. “Populism, PersistentRepublicanism and Declinism: An Empirical Analysis of Populismas a Thin Ideology.” Government and Opposition 51 (1): 111–33.
Erber, Ralph, and Richard R. Lau. 1990. “Political Cynicism Revis-ited: An Information-Processing Reconciliation of Policy-Basedand Incumbency-Based Interpretations of Changes in Trust inGovernment.” American Journal of Political Science 34 (1): 236.
Gerring, John. 1997. “Ideology: A Definitional Analysis.” PoliticalResearch Quarterly 50 (4): 957–94.
Gerring, John. 1999. “What Makes a Concept Good? A CriterialFramework for Understanding Concept Formation in the SocialSciences.” Polity 31 (3): 357–93.
Geurkink, Bram, Andrej Zaslove, Roderick Sluiter, and KristofJacobs. 2019. “Populist Attitudes, Political Trust, and ExternalPolitical Efficacy: Old Wine in New Bottles?” Political Studies 23(4): 003232171984276.
Goertz, Gary. 2006. Social Science Concepts: A User’s Guide.Princeton, NJ: Princeton University Press.
Goertz, Gary. 2020. Social Science Concepts and Measurement: Newand Completely Revised Edition. Princeton, NJ: Princeton Uni-versity Press.
Greco, Salvatore, Alessio Ishizaka, Menelaos Tasiou, and GianpieroTorrisi. 2019. “On the Methodological Framework of CompositeIndices: A Review of the Issues of Weighting, Aggregation, andRobustness.” Social Indicators Research 141 (1): 61–94.
Hameleers, Michael, Linda Bos, and Claes H. de Vreese. 2017. “TheAppeal of Media Populism: The Media Preferences of Citizens withPopulistAttitudes.”MassCommunication&Society 20 (4): 481–504.
Hawkins, Kirk A. 2009. “Is Chavez Populist? Measuring PopulistDiscourse in Comparative Perspective.” Comparative PoliticalStudies 42 (8): 1040–67.
Hawkins, Kirk A., and Cristobal Rovira Kaltwasser. 2018. “Intro-duction—Concept, Theory, and Method.” In The Ideational Ap-proach to Populism: Concept, Theory, and Analysis, eds. Kirk A.Hawkins, Ryan Carlin, Levente Littvay, and Cristobal RoviraKaltwasser. London: Routledge, 1–24.
Hawkins,KirkA.,RyanCarlin,LeventeLittvay, andCristobalRoviraKaltwasser, eds. 2018. The Ideational Approach to Populism:Concept, Theory, and Analysis. London: Routledge.
Hawkins, Kirk A., Scott Riding, and Cas Mudde. 2012. “MeasuringPopulist Attitudes.” IPSA Committee on Concepts and Methods.C&M Working Paper #55.
Hieda, Takeshi,MasahiroZenkyo, andMasaruNishikawa. 2019. “DoPopulists Support Populism? An Examination through an OnlineSurvey Following the 2017 Tokyo Metropolitan Assembly Elec-tion.” Party Politics 23 (4): 135406881984811.
Jansen, Robert S. 2011. “Populist Mobilization: A New TheoreticalApproach to Populism.” Sociological Theory 29 (2): 75–96.
Jungkunz, Sebastian. 2019. “Towards a Measurement of ExtremeLeft-Wing Attitudes.” German Politics 28 (1): 101–22.
Alexander Wuttke, Christian Schimpf, and Harald Schoen
372
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
Katz, Richard S. 1997. “Representational Roles.” European Journalof Political Research 32 (2): 211–26.
Kenny, Paul D. 2017. Patronage and Populism: Explaining PopulistSuccess in India,Asia, andBeyond. Oxford:OxfordUniversity Press.
Kinder, Donald R., and Nathan P. Kalmoe. 2017.Neither Liberal NorConservative: Ideological Innocence in the American Public. Uni-versity of Chicago Press.
Kolen, Michael J., and Robert L. Brennan. 2014. Test Equating,Scaling, and Linking: Methods and Practices. 3rd edition. Statisticsfor Social and Behavioral Sciences. New York, NY: Springer.
Laclau, Ernesto, and Chantal Mouffe. 1985. Hegemony and SocialistStrategy: Towards a Radical Democratic Politics. 2nd edition.London: Verso.
Liu, Chen-Wei, andR. PhilipChalmers. 2018. “Fitting ItemResponseUnfoldingModels toLikert-ScaleDataUsingMirt inR.”PLoSOne13 (5): e0196292.
Luskin, Robert C. 1987. “Measuring Political Sophistication.”American Journal of Political Science 31 (4): 856–99.
Mader, Matthias, Nils D. Steiner, and Harald Schoen. 2019. “TheGlobalisation Divide in the Public Mind: Belief Systems onGlobalisation and Their Electoral Consequences.” Journal ofEuropean Public Policy: 1–20. Published online 14 October 2019.
Mader, Matthias, Thomas J. Scotto, Jason Reifler, Peter H. Gries,Pierangelo Isernia, and Harald Schoen. 2018. “How Political AreNational Identities?AComparison of theUnitedStates, theUnitedKingdom, and Germany in the 2010s.” Research & Politics 5 (3).https://doi.org/10.1177/2053168018801469.
March, Luke. 2017. “Left andRight PopulismCompared: The BritishCase.” The British Journal of Politics & International Relations 19(2): 282–303.
Miller, Arthur H. 1974. “Political Issues and Trust in Government:1964–1970.” Annual Review of Political Science 68: 951–72.
Moffitt, Benjamin, and Simon Tormey. 2014. “Rethinking Populism:Politics, Mediatisation and Political Style.” Political Studies 62 (2):381–97.
Mohrenberg, Steffen, Robert A. Huber, and Tina Freyburg. 2019.“Love at First Sight? Populist Attitudes and Support for DirectDemocracy.” Party Politics. Published online 14 August 2019.
Møller, Jørgen, and Svend-Erik Skaaning. 2012. Requisites of De-mocracy: Conceptualization, Measurement, and Explanation. NewYork: Routledge.
Mudde, Cas. 2004. “The Populist Zeitgeist.” Government and Op-position 39 (4): 542–63.
Mudde, Cas. 2007. Populist Radical Right Parties in Europe. Cam-bridge and UK and New York: Cambridge University Press.
Mudde, Cas. 2017. “Populism: An Ideational Approach.” In TheOxfordHandbook of Populism. OxfordHandbook of Politics, eds.Cristobal Rovira Kaltwasser, Paul Taggart, Paulina O. Espejo, andPierre Ostiguy. New York NY: Oxford University Press, 27–47.
Mudde, Cas, and Cristobal Rovira Kaltwasser. 2013a. “Exclusionaryvs. Inclusionary Populism: Comparing Contemporary Europe andLatin America.” Government and Opposition 48 (2): 147–74.
Mudde, Cas, and Cristobal Rovira Kaltwasser. 2013b. “Populism.” InTheOxfordHandbook of Political Ideologies. OxfordHandbooks inPolitics & International Relations, eds. Michael Freeden, Lyman T.Sargent, andMarc Stears.Oxford:OxfordUniversity Press, 493–512.
Muller, Jan-Werner. 2017. “Populism and Constitutionalism.” In TheOxfordHandbook of Populism. OxfordHandbook of Politics, eds.Cristobal Rovira Kaltwasser, Paul Taggart, Paulina O. Espejo, andPierre Ostiguy. New York, NY: Oxford University Press, 590–606.
Munck, Gerardo L. 2009. Measuring Democracy: A Bridge betweenScholarship and Politics. Democratic Transition and Consolidation.Baltimore, Maryland: Johns Hopkins University Press.
Munck, Gerardo L., and Jay Verkuilen. 2002. “Conceptualizing andMeasuring Democracy: Evaluating Alternative Indices.” Com-parative Political Studies 35 (1): 5–34.
Murphy, Gregory L. 2002. The Big Book of Concepts. Bradford Bks.Cambridge, Mass: MIT Press.
Niemi, Richard G., Stephen C. Craig, and Franco Mattei. 1991.“Measuring InternalPoliticalEfficacy in the 1988NationalElectionStudy.” American Political Science Review 85 (4): 1407.
Oliver, J.Eric, andWendyM.Rahn.2016.“Riseof theTrumpenvolk.”The ANNALS of the American Academy of Political and SocialScience 667 (1): 189–206.
Pappas, Takis S. 2016. “Modern Populism: Research Advances,Conceptual and Methodological Pitfalls, and the Minimal Defini-tion.” Oxford Research Encyclopedia of Politics. DOI: 10.1093/acrefore/9780190228637.013.17.
Petty, Richard E., and Jon A. Krosnick. 1995. Attitude Strength:Antecedents and Consequences. New York: Psychology Press.
Putnam,Robert D. 2000.BowlingAlone: TheCollapse andRevival ofAmerican Community. New York: Simon & Schuster.
Quinlan, Stephen, and Deirdre Tinney. 2019. “A Populist Wave orMetamorphosisof aChameleon?PopulistAttitudesand theVote in2016 in theUnitedStatesand Ireland.”EconomicandSocialReview50 (2): 281–324.
Reeves, Sharon. L., C. Chaowadee Varakamin, and Christiani J.Henry. 1996. “The Relationship betweenArm-SpanMeasurementand Height with Special Reference to Gender and Ethnicity.”European Journal of Clinical Nutrition 50 (6): 398–400.
Rooduijn, Matthijs. 2014. “TheNucleus of Populism: In Search of theLowest Common Denominator.” Government and Opposition 49(4): 573–99.
Rooduijn, Matthijs. 2019. “State of the Field: How to Study Populismand Adjacent Topics? A Plea for Both More and Less Focus.”European Journal of Political Research 58 (1): 362–372.
Roßteutscher, Sigrid, Rudiger Schmitt-Beck, Harald Schoen, Bern-hard Weßels, Christof Wolf, Ina Bieber, Lars-ChristopherStovsand, Melanie Dietz, Philipp Scherer, AikoWagner, ReinholdMelcher,andHeikoGiebler. 2019.“ZA6802:Pre-andPost-electionCross Section (Cumulation) (GLES 2017): Version 3.0.1.” GESISData Archive.
Roßteutscher, Sigrid, Rudiger Schmitt-Beck, Harald Schoen, Bern-hard Weßels, Christof Wolf, Maria Preißinger, Agatha Kratz, Al-exander Wuttke, and Lea Gartner. 2018. Short-term CampaignPanel 2017 (GLES). Trans. University of Mannheim, Chair forPolitical Psychology. GESIS Datenarchiv. Cologne, ZA6804.
Sartori, Giovanni. 1970. “Concept Misformation in ComparativePolitics.” American Political Science Review 64 (4): 1033–53.
Sartori, Giovanni. 1984. “Guidelines for ConceptAnalysis.” In SocialScience Concepts: A Systematic Analysis, ed. Giovanni Sartori.Beverly Hills, California: Sage, 15–85.
Schedler, Andreas, and Rodolfo Sarsfield. 2007. “Democrats withAdjectives: Linking Direct and Indirect Measures of Demo-cratic Support.” European Journal of Political Research 46 (5):637–59.
Schulz,Anne, PhilippMuller, Christian Schemer,DominiqueS.Wirz,Wettstein Martin, and Werner Wirth. 2018. “Measuring PopulistAttitudes on Three Dimensions.” International Journal of PublicOpinion Research 30 (2): 316–26.
Spierings, Niels, and Andrej Zaslove. 2017. “Gender, Populist Atti-tudes, and Voting: Explaining the Gender Gap in Voting forPopulist Radical Right and Populist Radical Left Parties.” WestEuropean Politics 40 (4): 821–47.
Spruyt, Bram, Gil Keppens, and Filip van Droogenbroeck. 2016.“WhoSupportsPopulismandWhatAttractsPeople to It?”PoliticalResearch Quarterly 69 (2): 335–46.
Stanley, Ben. 2011. “Populism, Nationalism, or National Populism?AnAnalysis of SlovakVotingBehaviour at the 2010 ParliamentaryElection.”Communist and Post-Communist Studies 44 (4): 257–70.
Tsatsanis, Emmanouil, Ioannis Andreadis, and Eftichia Teperoglou.2018. “Populism from below: Socio-Economic and IdeologicalCorrelatesofMassAttitudes inGreece.”SouthEuropeanSociety&Politics 47 (9): 1–22.
Urbinati, Nadia. 2019. Me the People. Cambridge, MA: HarvardUniversity Press.
Van Hauwaert, Steven M., Christian H. Schimpf, and FlavioAzevedo. 2018. “Public opinion surveys: evaluating existingmeasures.” In The Ideational Approach to Populism: Concept,Theory, and Analysis, eds. Kirk A. Hawkins, Ryan Carlin,Levente Littvay, and Cristobal Rovira Kaltwasser. London:Routledge, 128–49.
VanHauwaert, StevenM.,ChristianH. Schimpf, andFlavioAzevedo.2019. “The Measurement of Populist Attitudes: Testing Cross-National Scales Using Item Response Theory.” Politics. Publishedonline 8 July 2019.
Van Hauwaert, Steven M., and Stijn van Kessel. 2018. “BeyondProtest and Discontent: A Cross-National Analysis of the Effect of
When the Whole Is Greater than the Sum of Its Parts
373
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
Populist Attitudes and Issue Positions on Populist Party Support.”European Journal of Political Research 57 (1): 68–92.
van Kessel, Stijn. 2015. Populist Parties in Europe: Agents of Dis-content? Basingstoke: Palgrave Macmillan.
Vehrkamp, Robert, and Christopher Wratil. 2017. A Populist Mo-ment? Populist Attitudes of Voters and Non-Voters before theGerman Federal Election 2017. Gutersloh: Bertelsmann.
Weyland,Kurt. 2001. “Clarifying aContestedConcept: Populism in theStudy of LatinAmerican Politics.”Comparative Politics 34 (1): 1–22.
Wong,Chi-Sum,Kenneth S. Law, andGuo-huaHuang. 2008. “On theImportance of Conducting Construct-Level Analysis for Multidi-mensional Constructs in Theory Development and Testing.”Journal of Management 34 (4): 744–64.
Wuttke,Alexander. 2019. “WhyTooMany Political Science FindingsCannot Be Trusted and What We Can Do About It: A Review ofMeta-Scientific Research and a Call for Academic Reform.”PolitischeVierteljahresschrift/GermanPolitical ScienceQuarterly60(1): 1–19.
Alexander Wuttke, Christian Schimpf, and Harald Schoen
374
Dow
nloa
ded
from
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e. IP
add
ress
: 54.
39.1
06.1
73, o
n 20
Aug
202
0 at
19:
29:4
2, s
ubje
ct to
the
Cam
brid
ge C
ore
term
s of
use
, ava
ilabl
e at
htt
ps://
ww
w.c
ambr
idge
.org
/cor
e/te
rms.
htt
ps://
doi.o
rg/1
0.10
17/S
0003
0554
1900
0807
