Adding Fuel to the Flames? Politicisation of EU Policy Evaluation … · 2019. 12. 20. · Adding Fuel to the Flames? Politicisation of EU Policy Evaluation in National Parliaments

CRIT ICAL PAPER

https://doi.org/10.1007/s11615-019-00213-6Polit Vierteljahresschr (2019) 60:805–821

Adding Fuel to the Flames? Politicisation of EU PolicyEvaluation in National Parliaments

Julian M. Hoerner

Published online: 12 December 2019© The Author(s) 2019

Abstract Under which conditions does national parliamentary discourse politiciseEuropean Union (EU) policy evaluation? In times of multiple crises and uncertainty,the alleged ‘democratic deficit’ of the EU, defined as an apparent lack of legiti-macy, has regained scholarly and popular attention. The European Commission andacademic commentators consider policy evaluation a specialised and targeted toolto improve the ‘output legitimacy’ of the EU by assessing policy effectiveness andefficiency. While evaluation can strengthen output legitimacy directly via learning,evaluation can be particularly effective when it becomes part of the national commu-nicative discourse on the EU. This discourse is most likely to take place in nationalparliaments, as they are the forums in which the government can be held to account.This paper relies on an automated content analysis of the share of keywords relatedto EU policy evaluation in debates in six national parliaments, covering a period of20 years. The findings show that the combination of popular and party Euroscepti-cism is crucial in determining parliamentary debate on EU policy evaluation. Pro-European parties generally do not refer to policy evaluation. However, if politi-cal parties are critical of European integration, EU policy evaluation is mentionedmore frequently. Under these conditions, members of parliaments also refer morefrequently to EU policy evaluation as the public becomes more Eurosceptic. Thesefindings suggest that EU policy evaluation is used as a tool for domestic politicalcontestation, with potential negative normative implications for the output legitimacyof the EU and for evaluation as a tool for learning.

Keywords EU policies · Legislative studies · European Union · Democraticdeficit · Euroscepticism

Electronic supplementary material The online version of this article (https://doi.org/10.1007/s11615-019-00213-6) contains supplementary material, which is available to authorized users.

J. M. Hoerner (�)London, United KingdomE-Mail: [email protected]

K

https://doi.org/10.1007/s11615-019-00213-6http://crossmark.crossref.org/dialog/?doi=10.1007/s11615-019-00213-6&domain=pdfhttp://orcid.org/0000-0003-2102-5613https://doi.org/10.1007/s11615-019-00213-6https://doi.org/10.1007/s11615-019-00213-6

806 J. M. Hoerner

Öl in die Flammen gießen? Politisierung von EU-Policy-Evaluation innationalen Parlamenten

Zusammenfassung Unter welchen Bedingungen wird die EU-Policy-EvaluationGegenstand von politischen Debatten in nationalen Parlamenten der EU-Mitglied-staaten? Vor dem Hintergrund des breit diskutierten Demokratiedefizits in der Eu-ropäischen Union wird die professionelle Evaluation von EU-Politiken sowohl vonder Europäischen Kommission selbst als auch von akademischen Beobachtern alsein mögliches Instrument zur Verbesserung der „output legitimacy“ der EU angese-hen. Evaluation kann zur direkten Verbesserung von Fachpolitiken durch Lerneffekteführen, gleichzeitig aber auch indirekt die Qualität nationaler Diskurse in konkretenPolitikfeldern steigern. Entsprechende Debatten sind insbesondere in den Plenen dernationalen Parlamente der EU-Mitgliedstaaten zu erwarten. Die vorliegende Studieanalysiert mithilfe eines computergestützten Textanalyseverfahrens Plenardebattenin sechs Mitgliedstaaten über einen Zeitraum von 20 Jahren. Die Befunde zeigen,dass Euroskeptizismus, sowohl in der Bevölkerung als auch auf der Ebene der Par-teien, die Häufigkeit der Nennung von Schlagwörtern mit Bezug zur Evaluationvon EU-Politiken stark beeinflusst. Abgeordnete pro-europäischer Parteien politi-sieren Evaluation nur selten, während mehr über Evaluation diskutiert wird, wenneuropaskeptische Parteien im Parlament vertreten sind. Insbesondere reagieren dieAbgeordneten unter diesen Bedingungen auf einen Anstieg des Euroskeptizismus inder Bevölkerung, indem sie mehr über Evaluation sprechen. Die Studie stellt daherfest, dass die Evaluation von EU-Politiken mitunter von Akteuren auf der nationa-len Ebene zu strategischen Zwecken im Parteienwettbewerb instrumentalisiert wird.Dies könnte sich potenziell negativ auf die Wirksamkeit der erhofften Lerneffektevon Evaluationsstudien und die Output-Legitimierung der EU auswirken.

Schlüsselwörter EU-Fachpolitiken · Legislative Studien · Europäische Union ·Demokratiedefizit · Euroskeptizismus

1 Introduction

In recent decades, the European Union has been affected by several crises relatingto highly salient policy areas, such as the common currency. These crises have hada profound impact on democratic outcomes and economic performance in manymember states, and thus the perceived problem-solving capacity of the EU. In thiscontext and against the background of long-standing debates on the alleged demo-cratic deficit, calls to improve the ‘output legitimacy’ of the EU have been voiced.The emphasis here lies on what the EU delivers to its citizens (Ward 2010, p. 123).Arguably, this form of legitimisation is particularly crucial to the EU multilevelsystem given the regulatory or efficiency-oriented focus of many policies (Majone1999). The importance for the EU of ‘Doing Less More Efficiently’ has been stressedby Commission President Juncker and by national politicians from across the po-litical spectrum, particularly following times of fiscal austerity in several memberstates (European Commission 2017).

K

Adding Fuel to the Flames? Politicisation of EU Policy Evaluation in National Parliaments 807

Policy evaluation is a potential tool to demonstrate the effectiveness, efficiency,and hence ‘output legitimacy’ of the EU. Evaluations provide detailed informationon the extent to which the actors involved have reached their policy goals (Corbettet al. 2011, p. 318–319 cited in Zwaan et al. 2016, p. 675). As such, evaluationhas a twofold aim: facilitating policy learning and demonstrating accountability(Versluis et al. 2011, p. 206). While learning can take place within an organisation,accountability requires exposure to a public forum (e.g. Zwaan et al. 2016, p. 678).Because output legitimacy of the multilevel system of the European Union hasto be demonstrated by national governments to their respective national publics,national parliaments are likely forums for such justification to take place. However,besides its main purpose of providing accountability and policy learning, evaluationcan also be used to manipulate political opportunity structures (Schoenefeld andJordan 2019, p. 372). By being made public—for example, by being discussed inthe plenary—policy evaluation becomes part of national parliamentary discourseand hence becomes politicised. The extent to which politicisation takes place haspotentially important normative and policy implications. Therefore, this paper aimsto answer the following research question: Under which conditions is EU policyevaluation mentioned in national parliamentary debates?

This paper uses dictionary-based computer-assisted content analysis of parlia-mentary debates to analyse the extent to which EU policy evaluation is debatedin the plenaries of the parliaments of Austria, France, Germany, Ireland, Spain,and the United Kingdom for the time period from the early 1990s until 2012. Thefindings suggest that, overall, Eurosceptic parties mention policy evaluation morefrequently than pro-European parties do. Furthermore, Eurosceptic public opinioncan lead to an increase in the number of mentions of EU policy evaluation. How-ever, the extent to which members of parliament (MPs) talk more or less aboutevaluation in reaction to changes in public opinion is moderated by the compositionof parliament. When parties in parliament are generally pro-European, an increasein public Euroscepticism as measured by Eurobarometer surveys does not have aneffect on the frequency with which EU policy evaluation is debated. However, whenissues entrepreneurs, parties that are Eurosceptic and for which Europe is salient (DeVries and Hobolt 2012), are strongly represented in parliament, policy evaluationis mentioned more frequently in reaction to Eurosceptic public opinion. Arguably,this implies that policy evaluation is not generally used by pro-European parties tounderline the output legitimacy of the EU in reaction to Eurosceptic public opinion,but only becomes part of parliamentary discourse when a Eurosceptic party uses itto criticise the EU. Pro-European parties might then be forced to refer to evaluationas well in response.

EU policy evaluation thus seems to be used as a politicised tool in domesticpolitical contestation. The findings have important normative implications. Con-cerns about how policy evaluation could be used strategically by political actorswere first voiced in the 1960s and 1970s. Weiss (1970) argues that with a growingnumber of social intervention programmes, evaluators would be subject to grow-ing political pressures, and evaluation results themselves might be used for partisanpurposes. Potentially, an increase in evaluation activity—intended as a means tocounter populist dynamics with technical arguments—might in fact help to further

K

808 J. M. Hoerner

undermine the output legitimacy of the EU if it becomes used in parliamentary dis-course by Eurosceptic parities. Moreover, this form of engagement with evaluationmight undermine the credibility of evaluation studies themselves, further weakeningevaluation as an instrument for accountability and learning.

The remainder of this paper is structured as follows: The next section elaborateson the role of policy evaluation in improving output legitimacy in the EU and itslink to accountability. The third section presents the hypotheses. The fourth sectiondescribes the paper’s chosen method and independent variables. It also explains therationale for the selection of country cases. The fifth section contains the modelspecification and analysis. The final section discusses the results and presents con-clusions.

2 Policy Evaluation, Output Legitimacy, and Accountability in theEuropean Union

The question of legitimacy of the multilevel system of the European Union hasbeen subject to much scholarly debate. At the core of discussions on the ‘demo-cratic deficit’ of the EU lies the assertion that national parliaments have lost powerin the process of European integration, which has strengthened national execu-tives—a weakness that the European Parliament supposedly is unable to compen-sate, even after its role has been strengthened in subsequent treaty reforms (e.g. asdiscussed in Moravcsik 1994). These critiques thus relate to a perceived lack of‘input legitimacy’ of the European Union, or the extent to which citizens participateand are adequately represented in the political process (Schmidt 2010, p. 17). More-over, the absence of a Europe-wide media, parties, and political competition moregenerally is said to contribute to this lack of legitimacy (e.g. Decker 2002; Follesdaland Hix 2006). Therefore, forms of legitimation that are not dependent on citizenparticipation and representation but rather on ‘the problem-solving capacity of themultilevel European polity’ gain particular importance, but are themselves under-mined by the institutional structures and decision-making rules of the EU (Scharpf2009, p. 198). Ward refers to output legitimacy as legitimacy “through the deliveryof results for the citizens” (2010, p. 123). This form of legitimacy is furthermoresalient given the historic core competencies of the EU in regulatory policy areas andtheir increased contestation (Schmidt 2010, p. 11). Thus, the legitimacy of the EUas an organisation with high consensus requirements relies to a large extent on thejustification of EU policies in the ‘communicative discourse’ in the national politicalarena (Scharpf 2009, p. 189).

Policy evaluation as an important factor in evidence-based policy making playsa crucial role in providing output legitimacy in the European Union (Widmer 2009).Indeed, the use of policy evaluation and the formalisation of evaluation practicehave increased in the European Union since the 1980s and 1990s (Widmer 2004,p. 35)—even though the coverage of legal acts and policy areas is by far not universal(Zwaan et al. 2016). Evaluation can contribute to output legitimacy of the EuropeanUnion via two mechanisms. Traditionally, evaluation has been seen as a tool forlearning or to improve the efficiency and effectiveness of EU policies (Versluis et al.

K


2011, p. 211–216). Most of the literature to date has focused on the role of policyevaluation in improving legitimacy in this context (e.g. Borras and Højlund 2015;Leeuw and Farubo 2008). However, there is a second avenue through which policyevaluation can have an impact on output legitimacy of the EU—and in particular thepublic perception thereof. The communication of the results of policy evaluation inpublic forums can reinforce the role of evaluation in influencing output legitimacyby making it part of the political discourse on the EU. This second mechanism is ofparticular importance in the EU context as output legitimacy does not rely merelyon ‘objective’ measures of policy success or failure, but also on the effectivenesswith which policies are justified by elites and the extent to which this discourseresonates with the citizens (Schmidt 2010, p. 7). In this context, mentions of policyevaluation can help to increase the perceived output legitimacy of the EuropeanUnion—in case evaluation is used by political actors to demonstrate the effectivenessand efficiency of EU policies, or can serve to undermine it—if parliamentary actorsrefer to evaluation studies to point out supposed failures of the EU multilevel systemin producing adequate policy outcomes. This public use of policy evaluation is thusclosely linked to the concept of accountability, i.e. a process whereby an agent isheld to account by its principal (e.g. Curtin et al. 2010). The role of policy evaluationin fostering accountability has been acknowledged (e.g. Versluis et al. 2011, p. 206)but has not been widely discussed or empirically tested in the literature (but seeZwaan et al. 2016).

Arguably, national parliaments are the most suitable institutions for the commu-nication and justification of EU policy outcomes. After all, national parliaments arethe venues in which national governments can be held publicly to account by theopposition and by their own backbenchers for activities in European affairs and fortheir voting behaviour in the Council of Ministers (e.g. Raunio and Hix 2000; Auelet al. 2015). Perhaps more so than the European Parliament, national legislaturesare focal points of (national) media attention and public interest in political dis-course (Auel and Raunio 2014; Hoerner 2017). In addition, Scharpf’s conceptionof discursive justification of policies focuses in particular on the justification of na-tional representatives in front of national publics, given the absence of a Europeandemos discussed above (Scharpf 2009, p. 173). Demonstrating and challenging thelegitimacy of the EU is thus a prime feature of national parliamentary activity inEU affairs. In this context, policy evaluation is arguably a highly pertinent tool inparliamentary discourse.

We would thus expect MPs to mention policy evaluation at least occasionallywhen debating the European Union and the value of the respective countries’ mem-bership in the latter. Discussing policy evaluation reports in the plenary would inpart be a strategy to demonstrate accountability, i.e. to demonstrate and justify thebehaviour of an agent (the European institutions) to the ultimate principal (the citi-zens) by referring to the findings of an independent actor (evaluator) along the chainof delegation in a public forum (the national parliament). Policy evaluation reportscould support this mechanism primarily by providing information, which is crucialto overcome the information deficit of the legislature vis-à-vis the government inEU affairs; such a deficit is considered one of the underlying reasons of the alleged

K

810 J. M. Hoerner

democratic deficit and the loss of influence of domestic legislatures (e.g. Moravcsik1994; Raunio and Hix 2000).

Under which conditions, then, is EU policy evaluation mentioned in the plenariesof national parliaments? And what can we infer from these patterns regarding thepurpose of its use? So far, only a few studies have focused on the politicisation ofpolicy evaluation in the form of parliamentary debates. Focusing on oral questionsin the European Parliament (EP), Zwaan et al. (2016) argue that members of theEuropean Parliament (MEPs) use references to evaluation primarily for agenda-setting purposes rather than to foster accountability. They find that interinstitutionalconflict between the European Commission and the EP is the best predictor ofa future reference to policy evaluation in a parliamentary question (Zwaan et al.2016, p. 688). Focusing on the use of reports of the European Court of Auditors inthe EP, Stephenson (2017, p. 47) argues that ‘MEPs are not objective evaluators; theyare biased politically-driven actors who may use ex-post evaluation to advance theirpolitical agendas and interests, often for short-term goals, including re-election’.Thus, strengthening the accountability mechanism might not be the only motivationfor MPs to refer to policy evaluation in the plenary. They might also want to use EUpolicy evaluation strategically, to signal a particular message about the EuropeanUnion to their constituents or to expose the position of the government or otherparties on the latter. European Union policy evaluation might become ‘politicised’,i.e. itself become part of a discourse brought about by ‘an increase in polarizationof opinions, interests or values and the extent to which they are publicly advancedtowards the process of policy formulation within the EU’ (De Wilde 2011, p. 560).

We would thus expect MPs to be reactive to public opinion when referring to EUpolicy evaluation and EU affairs more generally. Moreover, we could expect thispattern to be pronounced in national parliaments given their higher level of mediaexposure and more established electoral connection. Furthermore, party politicalconsiderations might play an important role in determining the MEPs’ willingnessto refer to policy evaluation when discussing the EU. The exact conditions underwhich MEPs could be expected to mention policy evaluation in the plenary arediscussed in the next section.

3 Hypotheses

In the past decades, the ‘permissive consensus’ said to have characterised Europeanintegration for many decades has given rise to a ‘constraining dissensus’ (Hoogheand Marks 2009, p. 13). Attitudes about the EU have become more critical, andcitizens’ assessments of the EU’s performance vary greatly across countries basedon a number of socio-economic factors and national benchmarking (De Vries 2018).Thus, if MPs refer to policy evaluation in an attempt as part of an accountabilitymechanism, we would expect to see a rise in the number of mentions of evalua-tion as a reaction to a rise in Eurosceptic public opinion. Evaluation provides theinformation necessary to demonstrate or challenge accountability, with the aim tohighlight or criticise the output legitimacy of the European Union. In particular,when public attitudes towards a country’s membership in the European Union are

K


negative, national elites could be more likely to refer to EU policy evaluation ina public parliamentary forum in order to underline or challenge the citizens’ per-ception of the output legitimacy of the European Union. We thus formulate thefollowing hypothesis:

H1: European Union evaluation is mentioned more frequently as public Euroscep-ticism increases.

However, besides the attitudes of the public towards the EU, the extent to whichMEPs refer to policy evaluation is also likely to be strongly influenced by partypolitical factors. The EU has become a contested issue in many countries in whichmost mainstream parties do not openly compete on European issues (Hooghe andMarks 2009, p. 10). Thus, the issue is often exploited by extremist parties on theleft and right (De Vries 2007, p. 267; Szczerbiak and Taggart 2008). These partiesthat seek to put an issue not extensively discussed previously onto the parliamentaryagenda have been termed ‘issue entrepreneurs’ (De Vries and Hobolt 2012, p. 250).Eurosceptic parties often focus on institutional or general EU issues rather thanon specific polity-orientated criticisms of the EU (Hoerner 2017; Senninger 2017).Hoerner finds that even when issue entrepreneurs scrutinise specific policies, theystill use this as an opportunity to criticise the EU at a more fundamental level (2017,p. 308). They might use policy evaluation to provide information on the weaknessesor failures of specific policies but relate those to more general critiques of the EUand its output legitimacy. Simultaneously, pro-European parties might be inclinedto make more reference to policy evaluation as well when issue entrepreneurs arepresent. As Eurosceptic issue entrepreneurs challenge the legitimacy of the EU, pro-European mainstream parties might in turn refer to policy evaluation to highlightthe output legitimacy of the European Union. Hence, a stronger presence of issueentrepreneurs might lead to more mentions of policy evaluation at the aggregatelevel.1 Thus:

H2: European Union evaluation is mentioned more frequently when issue en-trepreneurs are represented in parliament.

Moreover, we expect the presence of issue entrepreneurs to have a moderatingimpact on the effect of Euroscepticism. When faced with a Eurosceptic public, MPsof pro-European parties might want to avoid mentioning EU policy evaluation asthey face potentially high costs by debating technical issues on which they are likelyto diverge from their voters. By contrast, issue entrepreneurs might be particularlyincentivised to undermine the output legitimacy of the EU when public Euroscep-ticism is high. Moreover, mainstream parties might in turn be forced to refer to

1 The presence of issue entrepreneurs might thus have a direct effect—Eurosceptic parties themselves re-ferring more to critical findings of policy evaluation reports—and an indirect effect—pro-European partiesreferring more to policy success documented in evaluation reports to highlight the output legitimacy of theEU. The methodology of the current study does not allow us to distinguish between the relative importanceof these two pathways and between supportive and critical mentions of policy evaluation. However, thisaspect could be analysed in future research.

K

812 J. M. Hoerner

policy evaluation as well to bolster and underline the output legitimacy of the EU insuch a scenario. Overall, policy evaluation might thus be mentioned more frequentlywhen both the public and parliament are politicised on the issue of the EU. Thus:

H3: The effect of public Euroscepticism on the mentions of EU policy evaluationin the plenary is negative at low levels of issue entrepreneurship and positive at highlevels of issue entrepreneurship.

4 Method and Data

Parliamentary debates can be considered a crucial aspect of parliamentary activity inEU affairs because they fulfil a ‘communication function’ vis-à-vis the citizens (Aueland Raunio 2014, p. 2; Norton 1993). As such, they can also be considered an idealvenue to underline or challenge the output legitimacy of the EU by referring to EUpolicy evaluation. Legislative debates are generally a useful resource for researcherssince they are publicly available and MPs use them for a variety of purposes (Prokschand Slapin 2010, p. 335). Debates thus present an excellent opportunity to observedifferent preferences and emphases given by MPs to the EU in different countries ina highly visible forum. To assess how the independent variables impact the extentto which EU policy evaluation is debated in the plenary, a content analysis wasundertaken (Neuendorf 2016). The advantage of this approach is that it capturesmentions in all debates. The rationale is that if more EU evaluation keywords comeup in debates, this indicates that the parliament attributes more attention to the outputlegitimacy of the European Union.2

In order to improve comparability across countries and parliaments in whichdebates take place at different frequencies and different lengths (possibly for lin-guistic reasons), the proportion of evaluation keywords in all debates in a certainmonth was calculated instead of the proportion of evaluation keywords in individ-ual debates. For reasons of feasibility, two months per year were analysed: Marchand October. These months are characterised by strong parliamentary activity in allcountries under analysis, and usually no breaks take place in these months. The timeframe of the analysis at the level of parliaments is 1992 (ratification of the Maas-tricht Treaty) until 2012. This time frame was chosen because the Maastricht Treatyhas frequently been described as the starting point for significant politicisation andstronger European integration (Börzel and Risse 2009; Marks et al. 1996).

Two dictionaries were constructed for the present analysis: one containing evalu-ation keywords and one containing general keywords from a variety of policy areas(foreign affairs, taxes, etc.). The dictionaries were then applied to the documents foreach country/month using the programme QDA Miner/WordStat (Provalis Research,Montreal, Canada). Examples of the keywords can be found in the online Appendix.The proportion of all evaluation keywords of all keywords was then calculated andits logarithm used as the dependent variable.

2 The independent variables themselves are derived from a number of existing data sets.

K


The following countries were chosen as country cases for the analysis: Austria,Germany, France, Spain, Ireland, and the UK. These six countries represent an excel-lent institutional spread and the highest possible variation regarding the independentvariables of the study. The aim was to select a diverse set of cases (Gerring 2017,p. 97). Thus, the analysis includes countries with a very Eurosceptic electorate, suchas Austria and the UK, as well as countries with generally more pro-European vot-ers, such as Ireland. Moreover, countries with strong formal scrutiny powers, suchas Austria, and those with rather weak formal scrutiny powers, such as Ireland, areincluded (Winzen 2012). There is also strong variation regarding the average dissentwithin parties on European integration, with Austria and Germany showing verylow values and the UK with very high values. The same holds true for the presenceof Euroscepticism in the party system and the salience of the EU, as expressedby the issue-entrepreneurship score. Finally, the counties differ in their economictrajectories over the past decades, including the extent to which they were affectedby the Euro crisis. Arguably, this implies that the output legitimacy of the EU hasbeen challenged to varying degrees in the national communicative discourse.3

The independent variables to test the hypotheses are operationalised as follows:Euroscepticism is considered as the share of respondents in Eurobarometer surveyswho hold that the membership of their country is ‘a bad thing’ minus the share ofthose who think it is ‘a good thing’ (Eurobarometer, 2012). Issue entrepreneurshipis operationalised following the approach of De Vries and Hobolt (2012). The issue-entrepreneur score is generated by multiplying the salience score for each party inparliament with the sum of the mean party position of all parties in parliament onthe EU minus the party position of the party (De Vries and Hobolt 2012, p. 256).The salience score and the party position on European integration are both includedin the Chapel Hill Expert Survey (CHES) (Polk et al. 2017; Bakker et al. 2015).They are measured on score scales of 1–5 and 1–7, respectively, with higher valuesindicating higher salience and a more positive position on European integration,respectively. The distance between the position of a party on the EU and the meanparty position is thus negative when the party is more pro-European than the mean ofall parties and positive if it is more Eurosceptic (De Vries and Hobolt 2012, p. 256).For the analysis at the level of parliament, the sum of the issue-entrepreneur valuesfor all parties in parliament was calculated. The issue-entrepreneurship score usedhere is thus an aggregate measure and a continuous variable. The composition of thedifferent legislatures was derived from the ParlGov database (Döring and Manow2012).

The control variable concerning the formal rights of national parliaments in EUaffairs is based on Winzen’s data (2012, p. 663), which focus on information rights,the involvement of European Affairs Committees (EACs) and sectoral committees,and mandating rights.4 The strength of the parliaments’ formal powers in EU affairs

3 No country from Central or Eastern Europe was selected for this study. While this represents a potentiallimitation, dynamics of parliamentary control and evaluation systems still differ quite significantly fromother member states, making a comparison problematic.4 These scrutiny powers concern the extent to which the parliament is institutionally in a strong positionto hold its government to account, such as whether it is fully informed by the government of its actionsin EU affairs, whether there is a standing EU scrutiny committee, whether the government regularly sends

K

814 J. M. Hoerner

Table 1 Descriptive statistics

Count Mean Standarddeviation

Minimum Maximum

European Union evaluationkeywords (log)

206 –3.15 1.83 –7.59 0.14

Euroscepticism 213 –0.36 0.22 –0.80 0.11

Issue entrepreneurship 215 –1.38 2.07 –6.83 5.67

Formal rights 244 1.45 0.48 0.33 2.17

Council Presidency 252 0.06 0.24 0 1

Plenary agenda 252 1.17 0.90 0 2

Fig. 1 Change of share of evaluation keywords of all keywords over time

are included in the analysis because parliaments with a stronger role in EU affairsmight potentially be less reliant on information provided in the form of policyevaluation reports given their stronger independent access to relevant information.The variable for agenda control is based on Doering’s (1995, p. 225) parliamentaryagenda control index. The rationale for the inclusion of this control variable is thatlower levels of government control over the parliamentary agenda make it easier forbackbenchers and opposition MPs to hold the government to account, for exampleby mentioning critical policy evaluation studies. A dummy for whether the countryholds the Council Presidency at a given time is included. Arguably, MPs might

explanatory memoranda to parliament outlining its policy priorities, and whether there is a scrutiny reserveof mandating rights.

K


be particularly active in highlighting evaluation reports related to the Presidency’spriorities in plenary debates or call the government to action in certain areas. Finally,a monthly time-trend variable is included to account for a potential rise in evaluationoutput and activity over the period studied. The rationale here is that evaluation hasbecome more prominent in the EU policy process since the 1990s. In the absenceof data on the overall number of evaluation studies published per month, a lineartime-trend variable can be used as a proxy for an overall increase in evaluationactivity, which might drive an increase in references made to EU policy evaluationin national parliaments.

For the statistical analysis, a two-level random-intercept model was applied.A multilevel model was chosen given the highly structured nature of the data,with two monthly observations clustered in each country for each year. A laggeddependent variable was included in the model to account for temporal autocorrela-tion, as recommended by Becks and Katz (1995). Since the dependent variable wasa proportion and highly skewed towards zero, a log transformation was undertaken.Descriptive statistics can be found in Table 1.

As Fig. 1 shows, there is some interesting variation between countries and overtime. Overall, there seems to be a strong increase in mentions of policy evalua-tion over time—in line with the documented increase of evaluation activity in theEU. Furthermore, there seems to be a significantly lower level of mentions in Ire-land—perhaps a puzzling finding, as the country has benefited significantly fromEU structural funds, which were at the forefront of policies to be evaluated (e.g.Bachtler and Wren 2006). Nevertheless, the general pro-European attitude at thelevel of the public and of political parties as well as the status as a recipient country(and with strong support for EU-funded projects) might help to explain this pattern.

Table 2 Effects of theindependent variables on theshare of EU keywords of allwords at the parliamentary level

Coefficient SE

Euroscepticism 0.68 (0.76)

Issue entrepreneur 0.07** (0.04)

Interaction Euroscepticism x issue en-trepreneur

0.17*** (0.05)

Formal scrutiny powers 0.19 (0.20)

Agenda control 0.88 (0.57)

Council Presidency 0.32*** (0.12)

Lagged dependent variable 0.14* (0.08)

Time trend 0.003** (0.002)

Constant –5.53*** (1.07)

Countries 6 –

Observations 160 –

Standard errors (SE) clustered at the country level*Significant at the 0.1 level, **significant at the 0.05 level, ***signif-icant at the 0.01 level

K

816 J. M. Hoerner

Fig. 2 Marginal effect of Euroscepticism on the share of evaluation keywords at different values of theissue-entrepreneurship score

5 Analysis and Results

In the following, the main effects and the interaction of Euroscepticism and issueentrepreneurship are discussed in the order of the hypotheses. As the results of theanalysis in Table 2 show, the main effect for Euroscepticism is not significant atconventional levels of significance. This might indicate that MPs generally do notuniversally make reference to policy evaluation as a tool to highlight accountabilityand ‘output legitimacy’ of the EU when faced with a Eurosceptic public in orderto demonstrate accountability. However, an increase in the issue-entrepreneurshipscore is associated with a higher share of evaluation keywords, with a coefficient of0.07 significant at the 0.05 level. This indicates that Eurosceptic parties for whichEU affairs are salient are much more likely to refer to policy evaluation than pro-European parties are. This seems to indicate that policy evaluation is politicised, i.e.used as a political tool in partisan discourse when Eurosceptic parties are present. Asmentioned above, the underlying mechanism could be that Eurosceptic parties referto policy evaluation as an attempt to demonstrate weaknesses of EU policies—theyare more likely to use policy evaluation as a political tool. However, the currentmethod does not allow us to distinguish between the positive and negative mentionsof policy evaluation, so the overall increase of mentions of EU policy evaluationcould also be due to pro-European parties making more references when faced withEurosceptic public opinion. We now turn to the interaction of Euroscepticism andissue entrepreneurship.

K


Figure 2 shows the marginal effects for the interaction in terms of Euroscepticismand issue entrepreneurship. The y-axis shows the marginal effect of Euroscepticismon the share of evaluation keywords out of general dictionary keywords given theissue-entrepreneurship score of the party system. On the x-axis, higher values indi-cate a higher issue-entrepreneurship score of the party system. At the lowest level ofissue entrepreneurship—all parties represented in parliament are pro-European—theeffect of Euroscepticism is negative but not significant. This implies that pro-Euro-pean parties are not referring to policy evaluation in reaction to changes in publicopinion, for example to underline the output legitimacy of the European Union.However, at the highest level of issue entrepreneurship, an increase in Euroscepti-cism is associated with an increase in the share of evaluation keywords. The effect issignificant at the 0.05 level. Substantially, at the highest level of issue entrepreneur-ship, a 1% increase in public Euroscepticism leads to an increase in the share ofevaluation keywords of around 1.64%. Thus, policy evaluation seems to be referredto in national parliaments in reaction to an increase in Eurosceptic public opinionwhen (Eurosceptic) issue entrepreneurs are present. This finding further underlinesthe argument that policy evaluation is used as a politicised tool in national par-liaments. Rather than being used to underline the output legitimacy of the EU inreaction to critical public opinion, potentially by highlighting the effectiveness ofcertain policies, policy evaluation only becomes an element of parliamentary debateswhen it is activated in the presence of political actors—issue entrepreneurs—thatmight expect political gain from criticising EU policies against the backdrop of aEurosceptic public that is likely to be receptive to such arguments. Again, this trig-gering of evaluation mentions by issue entrepreneurs might also reinforce referencesby pro-European parties to neutralise the issue entrepreneurs’ claims—even thougha verification of this mechanism and the balance of positive and negative mentionsis beyond the scope of this study.

As for the control variables, neither formal scrutiny rights nor agenda controlpower of the parliaments are statistically significant. Thus, it does not seem to bethe case that MPs compensate for a lack of information on EU affairs received viainstitutionalised channels by referring to policy evaluation. Neither does it seem tomatter whether the government has a firm hold on the plenary agenda (for example,by preventing issue entrepreneurs from speaking on controversial issues). By con-trast, evaluation keywords seem to be mentioned more frequently when a countryholds the Council Presidency. Arguably, this might indicate that MPs may want tohighlight the EU’s performance in areas that are particularly salient for the govern-ment at the particular point in time, or to which it has committed publicly. However,the present approach cannot fully provide support for or against this claim. Finally,the linear time-trend variable is positive and significant at the 0.05 level—eventhough it is substantially small. This indicates that overall and controlling for thesubstantive variables of interest discussed above, the mentions of EU policy evalua-tion in the plenaries of the national parliaments studied here have increased in recentyears, perhaps as the result of an increase in the number of evaluations conducted.

K

818 J. M. Hoerner

6 Discussion and Conclusion

This paper has analysed the factors determining the extent to which EU policyevaluation is mentioned in debates in national parliaments. Theoretically, the paperstarted from the premise that output legitimacy—the effectiveness and efficiency ofEU policies—is particularly important in the European Union as more input-orientedpathways to provide legitimacy are arguably underdeveloped, following the classi-cal ‘democratic deficit’ argument. Policy evaluation can be seen as an importanttool to safeguard output legitimacy, for example by facilitating policy learning andimprovement. However, evaluation can also contribute to (perceived) output legiti-macy of the EU via a second pathway—when evaluation results are communicatedpublicly to the citizens. This use of evaluation is closely linked to accountability,as evaluation studies can provide information on EU policies that allow politicalactors to assess, challenge, or support their success or failure. Arguably, in a multi-level system such as the EU, the communication to national publics is particularlyimportant—especially given the alleged lack of a European demos. Thus, nationalparliaments are likely to be important venues in which output legitimacy can bedemonstrated or challenged in front of national publics, with members of parlia-ment holding the government to account over the EU’s performance. In this context,MPs might be reactive to public opinion on European integration to demonstrateor challenge output legitimacy in reaction to the preferences of the citizens as theirultimate principal.

Methodologically, this paper provided an original data set on the mentions ofEU policy-evaluation–related keywords in the plenaries of six Western Europeancountries, covering a 20-year period. The findings of the paper suggest that thementions of policy evaluation only increase in reaction to public opinion when is-sue entrepreneurs, Eurosceptic parties for which the EU is salient, are representedin parliament. When issue entrepreneurs are present, policy evaluation is generallymentioned more frequently than when parties are pro-European. The mechanismsbehind this finding might be that issue entrepreneurs themselves mention policyevaluation to criticise EU policies, pro-European parties are forced to underline out-put legitimacy of the EU by referring to evaluation studies, or there is a reinforcingcombination of the two pathways. This indicates that rather than being used as anaccountability tool to respond to citizens’ preferences in general, EU policy evalu-ation is used in national parliaments in a politicised way, with references being theresult of party–political considerations.

These findings have important implications for debates on policy evaluation ingeneral and use in a parliamentary context in particular. First, they demonstrate thatpolicy evaluation is indeed used as a political tool by political actors at the nationallevel. The potential for evaluation to demonstrate value for money, safeguard ac-countability, and foster policy learning has to be seen in a partisan context, withpolitical actors using evaluation studies strategically to react to and influence publicopinion. European Union policy evaluation is thus not only, or perhaps indeed pri-marily, used to demonstrate accountability to the citizens, but it becomes politicisedunder conditions of partisan conflict. Second, policy evaluation is particularly likelyto be referred to in the presence of issue entrepreneurs—parties that are often Eu-

K


rosceptic or populist. Policy evaluation as a ‘technocratic’ solution to problems ofaccountability and legitimacy in the European Union can thus also be understood asa tool that might be intended to demonstrate the output legitimacy of the EuropeanUnion but can also fuel populist and Eurosceptic critiques of the latter. This mightbe problematic from a normative point of view and might suggest that the use ofpolicy evaluation to highlight the output legitimacy of the EU bears potential risks.Smismans (2017) sees an increased politicisation of evaluation as problematic andwarns against attempting to use it as ‘a tool to appease populist discourse’ (p. 28).Eurosceptic and populist parties themselves could use evaluation findings to stressthe—from their perspective—excessive regulatory burden of the EU (ibid.). Thiscould not only further undermine the output legitimacy of the EU but might alsoerode the perception of policy evaluation as an objective, technical tool that providestrustworthy information on policy success or failure. A decline in the credibility ofevaluation could then also have negative consequences for the ability of EU policyevaluation to provide and facilitate policy learning. Arguably, this could be particu-larly problematic in times in which strategic disinformation and the trustworthinessof public statements are a major concern.

This study thus has made an original contribution to the analysis of the strategicuse and political contestation of EU policy evaluation. However, there are somelimitations of the present study. For example, the geographic and temporal scope ofthe study could be expanded by including more member states and using a longertime period. Moreover, while this study has shed light on the relationship betweenpolicy evaluation and public opinion in parliamentary debates, a number of questionsremain. For example, the current study was carried out at the level of parliaments,so it could not distinguish between different parties in terms of the frequency withwhich they mention EU policy evaluation. Moreover, future studies could analysethe extent to which positive and negative mentions of EU policy evaluation oc-cur, i.e. distinguish instances in which evaluation is used to support (supposedly bypro-European parties) or challenge (supposedly by issue entrepreneurs) the outputlegitimacy of the EU. Such an analysis could get a closer handle on the causal mech-anisms outlined in this paper and could be carried out either qualitatively or withmore advanced quantitative text-analysis methods. A number of other interestingquestions remain, such as looking at differences between policy areas and focusingon the use of evaluation in different venues (for example, in committees comparedto the plenary) or institutions (in the European Parliament compared to nationalparliaments). All of these points provide ample potential for future research at theintersection of public policy, legislative studies, and political behaviour.

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 Interna-tional License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution,and reproduction in any medium, provided you give appropriate credit to the original author(s) and thesource, provide a link to the Creative Commons license, and indicate if changes were made.

References

Auel, Katrin, and Raunio Tapio. 2014. Debating the state of the union? A comparative analysis of nationalparliaments in EU affairs. The Journal of Legislative Studies 20(1):13–28.

K

http://creativecommons.org/licenses/by/4.0/

820 J. M. Hoerner

Auel, Katrin, Olivier Rozenberg, and Angela Tacea. 2015. To scrutinise or not to scrutinise? Explainingvariation in EU-related activities in national parliaments. West European Politics 38(2):282–304.

Bachtler, John, and Colin Wren. 2006. Evaluation of European Union Cohesion policy: research questionsand policy challenges. Regional studies 40(2):143–153.

Bakker, Ryan, de Vries, Catherine, Edwards, Erica, Hooghe, Liesbet, Jolly, Seth, Marks, Gary, et al. 2015.Measuring party positions in Europe. Party Politics 21(1):143–152.

Becks, Nathaniel, and Jonathan Katz. 1995. What to do (and not to do) with time-series cross-section data.American Journal of Political Science 89(3):634–647.

Borras, Susana, and Steven Højlund. 2015. Evaluation and policy learning: the learner’s perspective. Eu-ropean Journal of Political Research 54:99–120.

Börzel, Tanja, and Thomas Risse. 2009. Revisiting the nature of the beast–politicization, European iden-tity, and postfunctionalism: a comment on Hooghe and Marks. British Journal of Political Science39(1):217–220.

Curtin, Deirdre, Mair, Peter, and Yannis Papadopoulos. 2010. Positioning accountability in European gov-ernance: an introduction. West European Politics 33(5):929–945.

Decker, Frank. 2002. Governance beyond the nation-state. Reflections on the democratic deficit of theEuropean Union. Journal of European Public Policy 9(2):256–272.

Döring, Herbert. 1995. Time as a scarce resource: government control of the agenda. In Parliaments andmajority rule in western Europe, ed. Herbert Döring, 223–246. New York: St. Martin’s Press.

Döring, Herbert, and Philipp Manow. 2012. Parliament and government composition database (ParlGov):An infrastructure for empirical information on parties, elections and governments in modern democ-racies. Version 12/10–15 October 2012. Available at www.parlgov.org.

Eurobarometer. 2012. Percentage of respondents who answered the question Generally speaking, do youthink that (your country’s) membership of the European Community (Common Market) is ...? with ‘abad thing’ in Austria, France, Germany, Ireland, Spain and the UK between 1991 and 2011. https://ec.europa.eu/commfrontoffice/publicopinionmobile/index.cfm/Chart/getChart/chartType/lineChart//themeKy/3/groupKy/3/savFile/10000/periodStart/1973-09-01%2000:00:00.0/periodEnd/2011-05-06%2000:00:00.0. Accessed 14 May 2019.

European Commission. 2017. Task force on subsidiarity, proportionality and doing less more effi-ciently. https://ec.europa.eu/commission/priorities/democratic-change/better-regulation/task-force-subsidiarity-proportionality-and-doing-less-more-efficiently_en. Accessed 14 May 2019.

Føllesdal, Andreas, and Simon Hix. 2006. Why There is a Democratic Deficit in the EU: A Response toMajone and Moravcsik. Journal of Common Market Studies 44(9):533–562.

Gerring, John. 2017. Case study research: principles and practices. Cambridge: Cambridge UniversityPress.

Hoerner, Julian. 2017. Real scrutiny or smoke and mirrors: the determinants and role of resolutions ofnational parliaments in European Union affairs. European Union Politics 18(2):307–322.

Hooghe, Lisbeth, and Gary Marks. 2009. A postfunctionalist theory of European integration: from permis-sive consensus to constraining dissensus. British Journal of Political Science 39(1):1–23.

Leeuw, Frans L., and Jan-Eric Farubo. 2008. Evaluation systems—what are they and why study them?Evaluation 14(2):157–169.

Majone, Giandomenico. 1999. The regulatory state and its legitimacy problems. West European Politics22(2):1–24.

Marks, Gary, Lisbeth Hooghe, and Kermit Blank. 1996. European integration from the 1980s: state-centricv. multi-level governance. Journal of Common Market Studies 34(3):341–378.

Moravcsik, Andrew. 1994. Why the European community strengthens the state: international cooperationand domestic politics. Center for European studies working paper series, Vol. 52, 513–553.

Neuendorf, Kimberley. 2016. The content analysis guidebook. London: Sage.Norton, Philipp. 1993. Does parliament matter? Upper Saddle River, NJ: Prentice Hall Europe.Polk, Jonathan, Rovny, Jan, Bakker, Ryan, Edwards, Erica, Hooghe, Liesbet, Jolly, Seth, et al. 2017. Ex-

plaining the salience of anti-elitism and reducing political corruption for political parties in Europewith the 2014 Chapel Hill Expert Survey data. Research & Politics 2017(January–March):1–9.

Proksch, Sven-Oliver, and Jonathan Slapin. 2010. Look who’s talking: parliamentary debate in the Euro-pean Union. European Union Politics 11:333–357.

Raunio, Tapio, and Simon Hix. 2000. Backbenchers learn to fight back: European integration and parlia-mentary government. West European Politics 23(4):142–168.

Scharpf, Fritz. 2009. Legitimacy in the multilevel European polity. European Political Science Review1(2):173–204.

K

http://www.parlgov.orghttps://ec.europa.eu/commfrontoffice/publicopinionmobile/index.cfm/Chart/getChart/chartType/lineChart//themeKy/3/groupKy/3/savFile/10000/periodStart/1973-09-01%2000:00:00.0/periodEnd/2011-05-06%2000:00:00.0https://ec.europa.eu/commfrontoffice/publicopinionmobile/index.cfm/Chart/getChart/chartType/lineChart//themeKy/3/groupKy/3/savFile/10000/periodStart/1973-09-01%2000:00:00.0/periodEnd/2011-05-06%2000:00:00.0https://ec.europa.eu/commfrontoffice/publicopinionmobile/index.cfm/Chart/getChart/chartType/lineChart//themeKy/3/groupKy/3/savFile/10000/periodStart/1973-09-01%2000:00:00.0/periodEnd/2011-05-06%2000:00:00.0https://ec.europa.eu/commfrontoffice/publicopinionmobile/index.cfm/Chart/getChart/chartType/lineChart//themeKy/3/groupKy/3/savFile/10000/periodStart/1973-09-01%2000:00:00.0/periodEnd/2011-05-06%2000:00:00.0https://ec.europa.eu/commission/priorities/democratic-change/better-regulation/task-force-subsidiarity-proportionality-and-doing-less-more-efficiently_enhttps://ec.europa.eu/commission/priorities/democratic-change/better-regulation/task-force-subsidiarity-proportionality-and-doing-less-more-efficiently_en


Schmidt, Vivien. 2010. Democracy and Legitimacy in the European Union Revisited: Output, Input andThroughput, KFG Working Paper Series, No. 21, November 2010, Kolleg-Forschergruppe (KFG)“The Transformative Power of Europe” Freie Universität Berlin. http://userpage.fu-berlin.de/kfgeu/kfgwp/wpseries/WorkingPaperKFG_21.pdf. Accessed 14 May 2019.

Schoenefeld, Jonas, and Andrew Jordan. 2019. Environmental policy evaluation in the EU: between learn-ing, accountability and political opportunities? Environmental Politics 28(2):365–384.

Senninger, Roman. 2017. Issue expansion and selective scrutiny – how opposition parties used parliamen-tary questions about the European Union in the national arena from 1973 to 2013. European UnionPolitics 18(2):283–306.

Smismans, Stijn. 2017. The politicization of ex post policy evaluation in the EU. European Journal of LawReform 19(1):97–120.

Stephenson, Paul. 2017. The European Parliament’s use of the European Court of Auditors’ special re-ports in the scrutiny of EU budgetary performance. In Scrutiny of EU Policies Contributions to theWorkshop Organised by the RSCAS Foundation, ed. Alfreado De Feo, Brigid Laffan, 44–51.

Szczerbiak, Aleks, and Taggart Paul (eds.). 2008. Opposing Europe. The comparative party politics ofEuroscepticsim. Oxford: Oxford University Press.

Versluis, Esther, Van Keulen, Mendeltje, and Paul Stephenson. 2011. Analyzing the European Union policyprocess. Gordonsville, VA: Macmillan International Higher Education. Red Globe Press.

De Vries, Catherine. 2007. Sleeping giant: fact or fairytale? How European integration affects nationalelections. European Union Politics 8(3):363–385.

De Vries, Catherine. 2018. Euroscepticism and the future of European integration. Oxford: Oxford Uni-versity Press.

De Vries, Catherine, and Sara Hobolt. 2012. When dimension collide: the electoral success of issue en-trepreneurs. European Union Politics 13(2):246–268.

Ward, Thomas. 2010. The European Union: a crisis of legitimacy? European view 9(1):115–127.Weiss, Carol. 1970. The Politicization of evaluation research. Journal of Social Issues 26(4):57–68.Widmer, Thomas. 2004. The development and status of evaluation standards in Western Europe. New

Directions for Evaluation 104:31–42.Widmer, Thomas. 2009. The contribution of evidence-based policy to the output-oriented legitimacy of the

state. Evidence & Policy 5(4):351–372.DeWilde, Pieter. 2011. No polity for old politics? A framework for analysing the politicization of European

integration. Journal of European Integration 33(5):559–579.Winzen, Thomas. 2012. National parliamentary control of European Union affairs: a cross national and

longitudinal comparison. West European Politics 35(3):657–672.Zwaan, Pieter, Stijn van Voort, and Ellen Mastenbroek. 2016. Ex post legislative evaluation in the European

Union: questioning the usage of evaluations as instruments for accountability. International Reviewof Administrative Science 82(6):674–693.

K

http://userpage.fu-berlin.de/kfgeu/kfgwp/wpseries/WorkingPaperKFG_21.pdfhttp://userpage.fu-berlin.de/kfgeu/kfgwp/wpseries/WorkingPaperKFG_21.pdf

Adding Fuel to the Flames? Politicisation of EU Policy Evaluation in National ParliamentsAbstractZusammenfassungIntroductionPolicy Evaluation, Output Legitimacy, and Accountability in the European UnionHypothesesMethod and DataAnalysis and ResultsDiscussion and ConclusionReferences

Documents

Adding Fuel to the Flames? Politicisation of EU Policy Evaluation … · 2019. 12. 20. · Adding Fuel to the Flames? Politicisation of EU Policy Evaluation in National Parliaments