Security Evaluation of PatternClassifiers under Attack

Battista Biggio, Member, IEEE , Giorgio Fumera, Member, IEEE , and Fabio Roli, Fellow, IEEE

Abstract—Pattern classification systems are commonly used in adversarial applications, like biometric authentication, network

intrusion detection, and spam filtering, in which data can be purposely manipulated by humans to undermine their operation. As this

adversarial scenario is not taken into account by classical design methods, pattern classification systems may exhibit vulnerabilities,

whose exploitation may severely affect their performance, and consequently limit their practical utility. Extending pattern classification

theory and design methods to adversarial settings is thus a novel and very relevant research direction, which has not yet been pursued

in a systematic way. In this paper, we address one of the main open issues: evaluating at design phase the security of pattern

classifiers, namely, the performance degradation under potential attacks they may incur during operation. We propose a framework for

empirical evaluation of classifier security that formalizes and generalizes the main ideas proposed in the literature, and give examples

of its use in three real applications. Reported results show that security evaluation can provide a more complete understanding of the

classifier’s behavior in adversarial environments, and lead to better design choices.

Index Terms—Pattern classification, adversarial classification, performance evaluation, security evaluation, robustness evaluation

Ç

1 INTRODUCTION

PATTERN classification systems based on machine learn-ing algorithms are commonly used in security-related

applications like biometric authentication, network intru-sion detection, and spam filtering, to discriminate betweena “legitimate” and a “malicious” pattern class (e.g., legiti-mate and spam emails). Contrary to traditional ones, theseapplications have an intrinsic adversarial nature since theinput data can be purposely manipulated by an intelligentand adaptive adversary to undermine classifier operation.This often gives rise to an arms race between the adversaryand the classifier designer. Well known examples ofattacks against pattern classifiers are: submitting a fakebiometric trait to a biometric authentication system(spoofing attack) [1], [2]; modifying network packetsbelonging to intrusive traffic to evade intrusion detectionsystems (IDSs) [3]; manipulating the content of spamemails to get them past spam filters (e.g., by misspellingcommon spam words to avoid their detection) [4], [5], [6].Adversarial scenarios can also occur in intelligent dataanalysis [7] and information retrieval [8]; e.g., a maliciouswebmaster may manipulate search engine rankings to arti-ficially promote her1 website.

It is now acknowledged that, since pattern classifica-tion systems based on classical theory and design meth-ods [9] do not take into account adversarial settings, theyexhibit vulnerabilities to several potential attacks, allow-ing adversaries to undermine their effectiveness [6], [10],[11], [12], [13], [14]. A systematic and unified treatment ofthis issue is thus needed to allow the trusted adoption ofpattern classifiers in adversarial environments, startingfrom the theoretical foundations up to novel design meth-ods, extending the classical design cycle of [9]. In particu-lar, three main open issues can be identified: (i) analyzingthe vulnerabilities of classification algorithms, and thecorresponding attacks [11], [14], [15]; (ii) developingnovel methods to assess classifier security against theseattacks, which is not possible using classical performanceevaluation methods [6], [12], [16], [17]; (iii) developingnovel design methods to guarantee classifier security inadversarial environments [1], [6], [10].

Although this emerging field is attracting growing inter-est [13], [18], [19], the above issues have only been sparselyaddressed under different perspectives and to a limitedextent. Most of the work has focused on application-specificissues related to spam filtering and network intrusion detec-tion, e.g., [3], [4], [5], [6], [10], while only a few theoreticalmodels of adversarial classification problems have beenproposed in the machine learning literature [10], [14], [15];however, they do not yet provide practical guidelines andtools for designers of pattern recognition systems.

Besides introducing these issues to the pattern recogni-tion research community, in this work we address issues (i)and (ii) above by developing a framework for the empiricalevaluation of classifier security at design phase that extendsthe model selection and performance evaluation steps of theclassical design cycle of [9].

In Section 2 we summarize previous work, and point outthree main ideas that emerge from it. We then formalizeand generalize them in our framework (Section 3). First, to

1. The adversary is hereafter referred to as feminine due to the pop-ular interpretation as “Eve” or “Carol” in cryptography and computersecurity.

� The authors are with the Department of Electrical and ElectronicEngineering, University of Cagliari, Piazza d’Armi, 09123 Cagliari, Italy.E-mail: {battista.biggio, fumera, roli}@diee.unica.it.

Manuscript received 24 July 2012; revised 28 Jan. 2013; accepted 27 Mar.2013; date of publication 4 Apr. 2013; date of current version 18 Mar. 2014.Recommended for acceptance by E. Ferrari.For information on obtaining reprints of this article, please send e-mail to:reprints@ieee.org, and reference the Digital Object Identifier below.Digital Object Identifier no. 10.1109/TKDE.2013.57

984 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 26, NO. 4, APRIL 2014

pursue security in the context of an arms race it is not suffi-cient to react to observed attacks, but it is also necessary toproactively anticipate the adversary by predicting the most rele-vant, potential attacks through a what-if analysis; this allowsone to develop suitable countermeasures before the attackactually occurs, according to the principle of security bydesign. Second, to provide practical guidelines for simulatingrealistic attack scenarios, we define a general model of theadversary, in terms of her goal, knowledge, and capability,which encompasses and generalizes models proposed inprevious work. Third, since the presence of carefully tar-geted attacks may affect the distribution of training and test-ing data separately, we propose a model of the datadistribution that can formally characterize this behavior, andthat allows us to take into account a large number of poten-tial attacks; we also propose an algorithm for the generationof training and testing sets to be used for security evaluation,which can naturally accommodate application-specific andheuristic techniques for simulating attacks.

In Section 4 we give three concrete examples of applica-tions of our framework in spam filtering, biometric authen-tication, and network intrusion detection. In Section 5, wediscuss how the classical design cycle of pattern classifiersshould be revised to take security into account. Finally, inSection 6, we summarize our contributions, the limitationsof our framework, and some open issues.

2 BACKGROUND AND PREVIOUS WORK

Here we review previous work, highlighting the conceptsthat will be exploited in our framework.

2.1 A Taxonomy of Attacks Against PatternClassifiers

A taxonomy of potential attacks against pattern classifierswas proposed in [11], [15], and subsequently extended in[14]. We will exploit it in our framework, as part of the defi-nition of attack scenarios. The taxonomy is based on twomain features: the kind of influence of attacks on the classi-fier, and the kind of security violation they cause. The influ-ence can be either causative, if it undermines the learningalgorithm to cause subsequent misclassifications; or explor-atory, if it exploits knowledge of the trained classifier tocause misclassifications, without affecting the learning algo-rithm. Thus, causative attacks may influence both trainingand testing data, or only training data, whereas exploratoryattacks affect only testing data. The security violation can bean integrity violation, if it allows the adversary to access theservice or resource protected by the classifier; an availabil-ity violation, if it denies legitimate users access to it; or a pri-vacy violation, if it allows the adversary to obtainconfidential information from the classifier. Integrity viola-tions result in misclassifying malicious samples as legiti-mate, while availability violations can also cause legitimatesamples to be misclassified as malicious. A third feature ofthe taxonomy is the specificity of an attack, that ranges fromtargeted to indiscriminate, depending on whether theattack focuses on a single or few specific samples (e.g., aspecific spam email misclassified as legitimate), or on awider set of samples.

2.2 Limitations of Classical Performance EvaluationMethods in Adversarial Classification

Classical performance evaluation methods, like k-fold crossvalidation and bootstrapping, aim to estimate the perfor-mance that a classifier will exhibit during operation, byusing data D collected during classifier design.2 These meth-ods are based on the stationarity assumption that the dataseen during operation follow the same distribution as D.Accordingly, they resample D to construct one or morepairs of training and testing sets that ideally follow thesame distribution as D [9]. However, the presence of anintelligent and adaptive adversary makes the classificationproblem highly non-stationary, and makes it difficult to pre-dict how many and which kinds of attacks a classifier willbe subject to during operation, that is, how the data distri-bution will change. In particular, the testing data processedby the trained classifier can be affected by both exploratoryand causative attacks, while the training data can only beaffected by causative attacks, if the classifier is retrainedonline [11], [14], [15].3 In both cases, during operation, test-ing data may follow a different distribution than that of train-ing data, when the classifier is under attack. Therefore,security evaluation cannot be carried out according to theclassical paradigm of performance evaluation.4

2.3 Arms Race and Security by Design

Security problems often lead to a “reactive” arms racebetween the adversary and the classifier designer. Ateach step, the adversary analyzes the classifier defenses,and develops an attack strategy to overcome them. Thedesigner reacts by analyzing the novel attack samples,and, if required, updates the classifier; typically, byretraining it on the new collected samples, and/or add-ing features that can detect the novel attacks (see Fig. 1,left). Examples of this arms race can be observed inspam filtering and malware detection, where it has ledto a considerable increase in the variability and sophisti-cation of attacks and countermeasures.

To secure a system, a common approach used in engi-neering and cryptography is security by obscurity, that relieson keeping secret some of the system details to the adver-sary. In contrast, the paradigm of security by design advo-cates that systems should be designed from the ground-upto be secure, without assuming that the adversary may everfind out some important system details. Accordingly, the

2. By design we refer to the classical steps of [9], which include fea-ture extraction, model selection, classifier training and performanceevaluation (testing). Operation denotes instead the phase which startswhen the deployed classifier is set to operate in a real environment.

3. “Training” data refers both to the data used by the learning algo-rithm during classifier design, coming from D, and to the data collectedduring operation to retrain the classifier through online learning algo-rithms. “Testing” data refers both to the data drawn from D to evaluateclassifier performance during design, and to the data classified duringoperation. The meaning will be clear from the context.

4. Classical performance evaluation is generally not suitable fornon-stationary, time-varying environments, where changes in the datadistribution can be hardly predicted; e.g., in the case of concept drift[20]. Further, the evaluation process has a different goal in this case,i.e., to assess whether the classifier can “recover” quickly after a changehas occurred in the data distribution. Instead, security evaluation aimsat identifying the most relevant attacks and threats that should be coun-tered before deploying the classifier (see Section 2.3).

system designer should anticipate the adversary by simulat-ing a “proactive” arms race to (i) figure out the most rele-vant threats and attacks, and (ii) devise propercountermeasures, before deploying the classifier (see Fig. 1,right). This paradigm typically improves security by delay-ing each step of the “reactive” arms race, as it requires theadversary to spend a greater effort (time, skills, and resour-ces) to find and exploit vulnerabilities. System securityshould thus be guaranteed for a longer time, with less fre-quent supervision or human intervention.

The goal of security evaluation is to address issue (i)above, i.e., to simulate a number of realistic attack scenariosthat may be incurred during operation, and to assess theimpact of the corresponding attacks on the targeted classi-fier to highlight the most critical vulnerabilities. Thisamounts to performing a what-if analysis [21], which is acommon practice in security. This approach has beenimplicitly followed in several previous works (see Sec-tion 2.4), but never formalized within a general frameworkfor the empirical evaluation of classifier security. Althoughsecurity evaluation may also suggest specific countermeas-ures, the design of secure classifiers, i.e., issue (ii) above,remains a distinct open problem.

2.4 Previous Work on Security Evaluation

Many authors implicitly performed security evaluation as awhat-if analysis, based on empirical simulation methods;however, they mainly focused on a specific application,classifier and attack, and devised ad hoc security evaluationprocedures based on the exploitation of problem knowledgeand heuristic techniques [1], [2], [3], [4], [5], [6], [15], [22],[23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34],[35], [36]. Their goal was either to point out a previouslyunknown vulnerability, or to evaluate security against aknown attack. In some cases, specific countermeasures werealso proposed, according to a proactive/security-by-designapproach. Attacks were simulated by manipulating trainingand testing samples according to application-specific crite-ria only, without reference to more general guidelines; con-sequently, such techniques cannot be directly exploited by asystem designer in more general cases.

A few works proposed analytical methods to evaluate thesecurity of learning algorithms or of some classes of deci-sion functions (e.g., linear ones), based on more general,application-independent criteria to model the adversary’sbehavior (including PAC learning and game theory) [10],[12], [14], [16], [17], [37], [38], [39], [40]. Some of these criteriawill be exploited in our framework for empirical securityevaluation; in particular, in the definition of the adversarymodel described in Section 3.1, as high-level guidelines forsimulating attacks.

2.5 Building on Previous Work

We summarize here the three main concepts more or lessexplicitly emerged from previous work that will beexploited in our framework for security evaluation.

1) Arms race and security by design: since it is not possibleto predict how many and which kinds of attacks a classifierwill incur during operation, classifier security should beproactively evaluated using a what-if analysis, by simulatingpotential attack scenarios.

2) Adversary modeling: effective simulation of attack sce-narios requires a formal model of the adversary.

3) Data distribution under attack: the distribution of testingdata may differ from that of training data, when the classi-fier is under attack.

3 A FRAMEWORK FOR EMPIRICAL EVALUATION OF

CLASSIFIER SECURITY

We propose here a framework for the empirical evaluationof classifier security in adversarial environments, that uni-fies and builds on the three concepts highlighted in Sec-tion 2.5. Our main goal is to provide a quantitative andgeneral-purpose basis for the application of the what-if anal-ysis to classifier security evaluation, based on the definitionof potential attack scenarios. To this end, we propose: (i) amodel of the adversary, that allows us to define any attackscenario; (ii) a corresponding model of the data distribution;and (iii) a method for generating training and testing setsthat are representative of the data distribution, and are usedfor empirical performance evaluation.

3.1 Attack Scenario and Model of the Adversary

Although the definition of attack scenarios is ultimately anapplication-specific issue, it is possible to give generalguidelines that can help the designer of a pattern recogni-tion system. Here we propose to specify the attack scenarioin terms of a conceptual model of the adversary that encom-passes, unifies, and extends different ideas from previouswork. Our model is based on the assumption that the adver-sary acts rationally to attain a given goal, according to herknowledge of the classifier, and her capability of manipulatingdata. This allows one to derive the corresponding optimalattack strategy.

Adversary’s goal. As in [17], it is formulated as the opti-mization of an objective function. We propose to definethis function based on the desired security violation(integrity, availability, or privacy), and on the attack specific-ity (from targeted to indiscriminate), according to the taxon-omy in [11], [14] (see Section 2.1). For instance, the goal ofan indiscriminate integrity violation may be to maximizethe fraction of misclassified malicious samples [6], [10],

Fig. 1. A conceptual representation of the arms race in adversarial classification. Left: the classical “reactive” arms race. The designer reacts to theattack by analyzing the attack’s effects and developing countermeasures. Right: the “proactive” arms race advocated in this paper. The designer triesto anticipate the adversary by simulating potential attacks, evaluating their effects, and developing countermeasures if necessary.

[14]; the goal of a targeted privacy violation may be toobtain some specific, confidential information from theclassifier (e.g., the biometric trait of a given user enrolledin a biometric system) by exploiting the class labelsassigned to some “query” samples, while minimizing thenumber of query samples that the adversary has to issueto violate privacy [14], [16], [41].

Adversary’s knowledge. Assumptions on the adversary’sknowledge have only been qualitatively discussed in pre-vious work, mainly depending on the application athand. Here we propose a more systematic scheme fortheir definition, with respect to the knowledge of the sin-gle components of a pattern classifier: (k.i) the trainingdata; (k.ii) the feature set; (k.iii) the learning algorithmand the kind of decision function (e.g., a linear SVM);(k.iv) the classifier’s decision function and its parameters(e.g., the feature weights of a linear classifier); (k.v) thefeedback available from the classifier, if any (e.g., the classlabels assigned to some “query” samples that the adver-sary issues to get feedback [14], [16], [41]). It is worth not-ing that realistic and minimal assumptions about whatcan be kept fully secret from the adversary should bedone, as discussed in [14]. Examples of adversary’sknowledge are given in Section 4.

Adversary’s capability. It refers to the control that theadversary has on training and testing data. We propose todefine it in terms of: (c.i) the attack influence (either causa-tive or exploratory), as defined in [11], [14]; (c.ii) whetherand to what extent the attack affects the class priors; (c.iii)how many and which training and testing samples can becontrolled by the adversary in each class; (c.iv) which fea-tures can be manipulated, and to what extent, taking intoaccount application-specific constraints (e.g., correlatedfeatures cannot be modified independently, and the func-tionality of malicious samples cannot be compromised [1],[3], [6], [10]).

Attack strategy. One can finally define the optimal attackstrategy, namely, how training and testing data should bequantitatively modified to optimize the objective functioncharacterizing the adversary’s goal. Such modifications aredefined in terms of: (a.i) how the class priors are modified;(a.ii) what fraction of samples of each class is affected by theattack; and (a.iii) how features are manipulated by theattack. Detailed examples are given in Section 4.

Once the attack scenario is defined in terms of the adver-sary model and the resulting attack strategy, our frameworkproceeds with the definition of the corresponding data dis-tribution, that is used to construct training and testing setsfor security evaluation.

3.2 A Model of the Data Distribution

We consider the standard setting for classifier design in aproblem which consists of discriminating between legiti-mate (L) and malicious (M) samples: a learning algorithmand a performance measure have been chosen, a set D ofn labelled samples has been collected, and a set of d fea-tures have been extracted, so that D ¼ fðxi; yiÞgni¼1, wherexi denotes a d-dimensional feature vector, and yi 2 fL;Mga class label. The pairs ðxi; yiÞ are assumed to be i.i.d.samples of some unknown distribution pDðX; Y Þ. Since

the adversary model in Section 3.1 requires us to specifyhow the attack affects the class priors and the features ofeach class, we consider the classical generative modelpDðX; Y Þ ¼ pDðY ÞpDðXjY Þ.5 To account for the presence ofattacks during operation, which may affect either thetraining or the testing data, or both, we denote the corre-sponding training and testing distributions as ptr and pts,respectively. We will just write p when we want to referto either of them, or both, and the meaning is clear fromthe context.

When a classifier is not under attack, according to theclassical stationarity assumption we have ptrðY Þ ¼ ptsðY Þ ¼pDðY Þ, and ptrðXjY Þ ¼ ptsðXjY Þ ¼ pDðXjY Þ. We extend thisassumption to the components of ptr and pts that are notaffected by the attack (if any), by assuming that they remainidentical to the corresponding distribution pD (e.g., if theattack does not affect the class priors, the above equalityalso holds under attack).

The distributions pðY Þ and pðXjY Þ that are affected bythe attack can be defined as follows, according to the defini-tion of the attack strategy, (a.i-iii).

Class priors. ptrðY Þ and ptsðY Þ can be immediately definedbased on assumption (a.i).

Class-conditional distributions. ptrðXjY Þ and ptsðXjY Þ canbe defined based on assumptions (a.ii-iii). First, to accountfor the fact that the attack may not modify all training ortesting samples, according to (a.ii), we model pðXjY Þ as amixture controlled by a Boolean random variable A, whichdenotes whether a given sample has been manipulated(A ¼ T) or not (A ¼ F):

pðXjY Þ ¼ pðXjY ;A ¼ TÞpðA ¼ TjY Þþ pðXjY ;A ¼ FÞpðA ¼ FjY Þ:

We name the samples of the component pðXjY ;A ¼ TÞattack samples to emphasize that their distribution is differ-ent from that of samples which have not been manipulatedby the adversary, pðXjY ;A ¼ FÞ. Note that pðA ¼ TjY ¼ yÞis the probability that an attack sample belongs to class y,i.e., the percentage of samples of class y controlled by theadversary. It is thus defined by (a.ii).

For samples which are unaffected by the attack, the sta-tionarity assumption holds:

pðXjY ;A ¼ FÞ ¼ pDðXjY Þ: (2)

The distribution pðXjY ;A ¼ TÞ depends on assumption(a.iii). Depending on the problem at hand, it may not be pos-sible to analytically define it. Nevertheless, for the purposeof security evaluation, we will show in Section 3.3 that itcan be defined as the empirical distribution of a set of ficti-tious attack samples.

Finally, pðXjY ;A ¼ FÞ can be defined similarly, if its ana-lytical definition is not possible. In particular, according toEq. (2), it can be defined as the empirical distribution of D.

The above generative model of the training and test-ing distributions ptr and pts is represented by the

5. In this paper, for the sake of simplicity, we use the lowercasenotation pð�Þ to denote any probability density function, with both con-tinuous and discrete random variables.

Bayesian network in Fig. 2, which corresponds to factor-ize pðX; Y ; AÞ as follows:

pðX; Y ; AÞ ¼ pðY ÞpðAjY ÞpðXjY ;AÞ: (3)

Our model can be easily extended to take into accountconcurrent attacks involving m > 1 different kinds of sam-ple manipulations; for example, to model attacks againstdifferent classifiers in multimodal biometric systems [1], [2].To this end, one can define m different Boolean random var-iables A ¼ ðA1; . . . ; AmÞ, and the corresponding distribu-tions. The extension of Eq. (3) is then straightforward (e.g.,see [32]). The dependence between the different attacks, ifany, can be modeled by the distribution pðAjY Þ.

If one is also interested in evaluating classifier securitysubject to temporal, non-adversarial variations of the datadistribution (e.g., the drift of the content of legitimate emailsover time), it can be assumed that the distributionpðX; Y ; A ¼ FÞ changes over time, according to some modelpðX; Y ; A ¼ F; tÞ (see, e.g., [20]). Then, classifier security attime t can be evaluated by modeling the attack distributionover time pðX; Y ; A ¼ T; tÞ as a function of pðX; Y ; A ¼ F; tÞ.

The model of the adversary in Section 3.1, and the abovemodel of the data distribution provide a quantitative, well-grounded and general-purpose basis for the application ofthe what-if analysis to classifier security evaluation, whichadvances previous work.

3.3 Training and Testing Set Generation

Here we propose an algorithm to sample training (TR) andtesting (TS) sets of any desired size from the distributionsptrðX; Y Þ and ptsðX; Y Þ.

We assume that k � 1 different pairs of training and test-ing sets ðDiTR;DiTSÞ, i ¼ 1; . . . ; k, have been obtained from Dusing a classical resampling technique, like cross-validationor bootstrapping. Accordingly, their samples follow the dis-tribution pDðX; Y Þ. In the following, we describe how tomodify each of the sets DiTR to construct a training set TRi

that follows the distribution ptrðX; Y Þ. For the sake of sim-plicity, we will omit the superscript i. An identical proce-dure can be followed to construct a testing set TSi from eachof the DiTS. Security evaluation is then carried out with theclassical method, by averaging (if k > 1) the performance ofthe classifier trained on TRi and tested on TSi.

If the attack does not affect the training samples, i.e.,ptrðX; Y Þ ¼ pDðX; Y Þ, TR is simply set equal to DTR. Other-wise, two alternatives are possible. (i) If ptrðXjY ;AÞ is ana-lytically defined for each Y 2 fL;Mg and A 2 fT;Fg, thenTR can be obtained by sampling the generative model ofpðX; Y ; AÞ of Eq. (3): first, a class label y is sampled fromptrðY Þ, then, a value a from ptrðAjY ¼ yÞ, and, finally, a fea-ture vector x from ptrðXjY ¼ y;A ¼ aÞ. (ii) If ptrðXjY ¼y;A ¼ aÞ is not analytically defined for some y and a, but aset of its samples is available, denoted in the following as

Dy;aTR, it can be approximated as the empirical distributionof Dy;aTR. Accordingly, we can sample with replacement fromDy;aTR [42]. An identical procedure can be used to constructthe testing set TS. The procedure to obtain TR or TS is for-mally described as Algorithm 1.6

Let us now discuss how to construct the sets Dy;aTR, when

ptrðXjY ¼ y;A ¼ aÞ is not analytically defined. The same dis-

cussion holds for the sets Dy;aTS. First, the two sets DL;FTR and

DM;FTR can be respectively set equal to the legitimate and mali-

cious samples in DTR, since the distribution of such samplesis assumed to be just ptrðXjY ¼ L; A ¼ FÞ and ptrðXjY ¼M; A ¼ FÞ (see Eq. (2)):DL;F

TR ¼ fðx; yÞ 2 DTR : y ¼ Lg,DM;FTR ¼

fðx; yÞ 2 DTR : y ¼ Mg. The two sets of attack samples Dy;TTR,

for y ¼ L;M, must come instead from ptrðXjY ¼ y;A ¼ TÞ.They can thus be constructed according to the point (a.iii) ofthe attack strategy, using any technique for simulating attacksamples. Therefore, all the ad hoc techniques used in previ-ous work for constructing fictitious attack samples can beused as methods to define or empirically approximate thedistribution pðXjY ¼ y; A ¼ TÞ, for y ¼ L;M.

Two further considerations must be made on the simu-lated attack samples Dy;TTR. First, the number of distinctattack samples that can be obtained depends on the simula-tion technique, when it requires the use of the data in DTR;for instance, if it consists of modifying deterministically eachmalicious sample in DTR, no more than jfðx; yÞ 2 DTR : y ¼Mgj distinct attack samples can be obtained. If any numberof attack samples can be generated, instead (e.g., [5], [28],[34], [36]), then the sets Dy;TTR , for y ¼ L;M, do not need tobe constructed beforehand: a single attack sample can be

Fig. 2. Generative model of the distributions ptr and pts.

6. Since the proposed algorithm is based on classical resamplingtechniques such as cross-validation and bootstrapping, it is reasonableto expect that the bias and the variance of the estimated classificationerror (or of any other performance measure) will enjoy similar statisti-cal properties to those exhibited by classical performance evaluationmethods based on the same techniques. In practice, these error compo-nents are typically negligible with respect to errors introduced by theuse of limited training data, biased learning/classification algorithms,and noisy or corrupted data.

generated online at step 1 of Algorithm 1, when a ¼ T. Sec-ond, in some cases it may be necessary to construct theattack samples incrementally; for instance, in the causativeattacks considered in [11], [30], [36], the attack samples areadded to the training data one at a time, since each of themis a function of the current training set. This corresponds toa non-i.i.d. sampling of the attack distribution. In thesecases, Algorithm 1 can be modified by first generating allpairs y; a, then, the feature vectors x corresponding toa ¼ F, and, lastly, the attack samples corresponding toa ¼ T, one at a time.

3.4 How to Use Our Framework

We summarize here the steps that the designer of a pat-tern classifier should take to evaluate its security usingour framework, for each attack scenario of interest. Theyextend the performance evaluation step of the classicaldesign cycle of [9], which is used as part of the modelselection phase, and to evaluate the final classifier to bedeployed.

1) Attack scenario. The attack scenario should be definedat the conceptual level by making specific assumptions onthe goal, knowledge (k.i-v), and capability of the adversary(c.i-iv), and defining the corresponding attack strategy (a.i-iii), according to the model of Section 3.1.

2) Data model. According to the hypothesized attack sce-nario, the designer should define the distributions pðY Þ,pðAjY Þ, and pðXjY ;AÞ, for Y 2 fL;Mg, A 2 fF;Tg, and fortraining and testing data. If pðXjY ;AÞ is not analyticallydefined for some Y ¼ y and A ¼ a, either for training or test-ing data, the corresponding set Dy;aTR or Dy;aTS must be con-

structed. The sets Dy;FTR (Dy;FTS ) are obtained from DTR (DTS).

The sets Dy;TTR and Dy;TTS can be generated, only if the attack

involves sample manipulation, using an attack sample sim-ulation technique according to the attack strategy (a.iii).

3) Construction of TR and TS. Given k � 1 pairsðDiTR;DiTSÞ, i ¼ 1; . . . ; k, obtained from classical resamplingtechniques like cross-validation or bootstrapping, the size ofTR and TS must be defined, and Algorithm 1 must be runwith the corresponding inputs to obtain TRi and TSi. If theattack does not affect the training (testing) data, TRi (TSi) isset to DiTR (DiTS).

4) Performance evaluation. The classifier performanceunder the simulated attack is evaluated using the con-structed (TRi, TSi) pairs, as in classical techniques.

4 APPLICATION EXAMPLES

While previous work focused on a single application, weconsider here three different application examples of ourframework in spam filtering, biometric authentication,and network intrusion detection. Our aim is to showhow the designer of a pattern classifier can use ourframework, and what kind of additional information hecan obtain from security evaluation. We will show that atradeoff between classifier accuracy and security emergessometimes, and that this information can be exploited forseveral purposes; e.g., to improve the model selectionphase by considering both classification accuracy andsecurity.

4.1 Spam Filtering

Assume that a classifier has to discriminate between legiti-mate and spam emails on the basis of their textual content,and that the bag-of-words feature representation has beenchosen, with binary features denoting the occurrence of agiven set of words. This kind of classifier has been consid-ered by several authors [6], [28], [43], and it is included inseveral real spam filters.7

In this example, we focus on model selection. Weassume that the designer wants to choose between a sup-port vector machine (SVM) with a linear kernel, and alogistic regression (LR) linear classifier. He also wants tochoose a feature subset, among all the words occurring intraining emails. A set D of legitimate and spam emails isavailable for this purpose. We assume that the designerwants to evaluate not only classifier accuracy in theabsence of attacks, as in the classical design scenario, butalso its security against the well-known bad word obfus-cation (BWO) and good word insertion (GWI) attacks.They consist of modifying spam emails by inserting“good words” that are likely to appear in legitimateemails, and by obfuscating “bad words” that are typicallypresent in spam [6]. The attack scenario can be modeledas follows.

1) Attack scenario. Goal. The adversary aims at maximiz-ing the percentage of spam emails misclassified as legiti-mate, which is an indiscriminate integrity violation.

Knowledge. As in [6], [10], the adversary is assumed tohave perfect knowledge of the classifier, i.e.,: (k.ii) the fea-ture set, (k.iii) the kind of decision function, and (k.iv) itsparameters (the weight assigned to each feature, and thedecision threshold). Assumptions on the knowledge of (k.i)the training data and (k.v) feedback from the classifier arenot relevant in this case, as they do not provide any addi-tional information.

Capability. We assume that the adversary: (c.i) is only ableto influence testing data (exploratory attack); (c.ii) cannotmodify the class priors; (c.iii) can manipulate each malicioussample, but no legitimate ones; (c.iv) can manipulate anyfeature value (i.e., she can insert or obfuscate any word),but up to a maximum number nmax of features in each spamemail [6], [10]. This allows us to evaluate how gracefully theclassifier performance degrades as an increasing number offeatures is modified, by repeating the evaluation for increas-ing values of nmax.

Attack strategy. Without loss of generality, let us furtherassume that x is classified as legitimate if gðxÞ ¼Pn

i¼1 wixi þ w0 < 0, where gð�Þ is the discriminant functionof the classifier, n is the feature set size, xi 2 f0; 1g are thefeature values (1 and 0 denote respectively the presence andthe absence of the corresponding term), wi are the featureweights, and w0 is the bias.

Under the above assumptions, the optimal attack strat-egy can be attained by: (a.i) leaving the class priorsunchanged; (a.ii) manipulating all testing spam emails; and(a.iii) modifying up to nmax words in each spam to minimizethe discriminant function of the classifier.

7. SpamAssassin, http://spamassassin.apache.org/; Bogofilter,http://bogofilter.sourceforge.net/; SpamBayes http://spambayes.sourceforge.net/.

Each attack sample (i.e., modified spam) can be thusobtained by solving a constrained optimization problem. Asin [10], the generation of attack samples can be representedby a function A : X 7!X , and the number of modified wordscan be evaluated by the Hamming distance. Accordingly,for any given x 2 X , the optimal attack strategy amounts tofinding the attack sample AðxÞ which minimizes gðAðxÞÞ,subject to the constraint that the Hamming distancebetween x and AðxÞ is no greater than nmax, that is:

AðxÞ ¼ argminx0

Xn

i¼1

wix0i

s:t:Xn

i¼1

jx0i � xij � nmax:

ð4Þ

The solution to problem (4) is straightforward. First, notethat inserting (obfuscating) a word results in switching thecorresponding feature value from 0 to 1 (1 to 0), and thatthe minimum of gð�Þ is attained when all features that havebeen assigned a negative (positive) weight are equal to 1 (0).Accordingly, for a given x, the largest decrease of gðAðxÞÞsubject to the above constraint is obtained by analyzing allfeatures for decreasing values of jwij, and switching from 0to 1 (from 1 to 0) the ones corresponding to wi < 0 (wi > 0),until nmax changes have been made, or all features havebeen analyzed. Note that this attack can be simulated bydirectly manipulating the feature vectors of spam emailsinstead of the emails themselves.

2) Data model. Since the adversary can only manipulatetesting data, we set ptrðY Þ ¼ pDðY Þ, and ptrðXjY Þ ¼ pDðXjY Þ.Assumptions (a.i-ii) are directly encoded as: (a.i)ptsðY Þ ¼ pDðY Þ; and (a.ii) ptsðA ¼ TjY ¼ MÞ ¼ 1, andptsðA ¼ TjY ¼ LÞ ¼ 0. The latter implies that ptsðXjY ¼LÞ ¼ pDðXjY ¼ LÞ. Assumption (a.iii) is encoded into theabove function AðxÞ, which empirically defines the distribu-tion of the attack samples ptsðXjY ¼ M; A ¼ TÞ. Similarly,pðXjY ¼ y;A ¼ FÞ ¼ pDðXjyÞ, for y ¼ L;M, will be approxi-mated as the empirical distribution of the set Dy;FTS obtainedfrom DTS, as described below; the same approximation willbe made in all the considered application examples.

The definition of the attack scenario and the data modelis summarized in Table 1 (first column).

3) Construction of TR and TS. We use a publicly availableemail corpus, TREC 2007. It consists of 25,220 legitimateand 50,199 real spam emails.8 We select the first 20,000emails in chronological order to create the data set D. Wethen split D into two subsets DTR and DTS, respectivelymade up of the first 10,000 and the next 10,000 emails, inchronological order. We use DTR to construct TR and DTS toconstruct TS, as in [5], [6]. Since the considered attack doesnot affect training samples, we set TR ¼ DTR. Then, we

define DL;FTS and DM;F

TS respectively as the legitimate and

malicious samples in DTS, and construct the set of attack

samples DM;TTS by modifying all the samples in DM;F

TS accord-

ing to AðxÞ, for any fixed nmax value. Since the attack does

not affect the legitimate samples, we set DL;TTS ¼ ;. Finally,

we set the size of TS to 10,000, and generate TS by running

Algorithm 1 on DL;FTS , DM;F

TS and DM;TTS .

The features (words) are extracted from TR using theSpamAssassin tokenization method. Four feature subsetswith size 1,000, 2,000, 10,000 and 20,000 have been selectedusing the information gain criterion [44].

4) Performance evaluation. The performance measure weuse is the area under the receiver operating characteristiccurve (AUC) corresponding to false positive (FP) error rates

in the range ½0; 0:1� [6]: AUC10% ¼R 0:1

0 TPðFPÞdFP 2 ½0; 0:1�,where TP is the true positive error rate. It is suited to classi-fication tasks like spam filtering, where FP errors (i.e., legiti-mate emails misclassified as spam) are much more harmfulthan false negative (FN) ones.

Under the above model selection setting (two classifiers,and four feature subsets) eight different classifier modelsmust be evaluated. Each model is trained on TR. SVMs areimplemented with the LibSVM software [45]. The C param-eter of their learning algorithm is chosen by maximizing theAUC10 percent through a 5-fold cross-validation on TR. Anonline gradient descent algorithm is used for LR. After clas-sifier training, the AUC10 percent value is assessed on TS, fordifferent values of nmax. In this case, it is a monotonicallydecreasing function of nmax. The more graceful its decrease,the more robust the classifier is to the considered attack.Note that, for nmax ¼ 0, no attack samples are included inthe testing set: the corresponding AUC10 percent value equalsthat attained by classical performance evaluation methods.

The results are reported in Fig. 3. As expected, theAUC10 percent of each model decreases as nmax increases. Itdrops to zero for nmax values between 30 and 50 (dependingon the classifier): this means that all testing spam emails gotmisclassified as legitimate, after adding or obfuscating from30 to 50 words.

The SVM and LR classifiers perform very similarly whenthey are not under attack (i.e., for nmax ¼ 0), regardless ofthe feature set size; therefore, according to the viewpoint ofclassical performance evaluation, the designer could chooseany of the eight models. However, security evaluation

TABLE 1Parameters of the Attack Scenario and of the Data Model for

Each Application Example

‘Empirical’ means that pðXjY ¼ y;A ¼ aÞ was approximated as theempirical distribution of a set of samples Dy;a; and ‘�’ means that nosamples from this distribution were needed to simulate the attack.

8. http://plg.uwaterloo.ca/~gvcormac/treccorpus07.

highlights that they exhibit a very different robustness tothe considered attack, since their AUC10 percent valuedecreases at very different rates as nmax increases; in partic-ular, the LR classifier with 20,000 features clearly outper-forms all the other ones, for all nmax values. This resultsuggests the designer a very different choice than the onecoming from classical performance evaluation: the LR clas-sifier with 20,000 features should be selected, given that itexhibit the same accuracy as the other ones in the absence ofattacks, and a higher security under the considered attack.

4.2 Biometric Authentication

Multimodal biometric systems for personal identity recog-nition have received great interest in the past few years. Ithas been shown that combining information coming fromdifferent biometric traits can overcome the limits and theweaknesses inherent in every individual biometric, result-ing in a higher accuracy. Moreover, it is commonly believedthat multimodal systems also improve security againstspoofing attacks, which consist of claiming a false identityand submitting at least one fake biometric trait to the system(e.g., a “gummy” fingerprint or a photograph of a user’sface). The reason is that, to evade a multimodal system, oneexpects that the adversary should spoof all the correspond-ing biometric traits. In this application example, we showhow the designer of a multimodal system can verify if thishypothesis holds, before deploying the system, by simulat-ing spoofing attacks against each of the matchers. To thisend, we partially exploit the analysis in [1], [2].

We consider a typical multimodal system, made up of afingerprint and a face matcher, which operates as follows.

The design phase includes the enrollment of authorizedusers (clients): reference templates of their biometric traitsare stored into a database, together with the correspondingidentities. During operation, each user provides therequested biometric traits to the sensors, and claims theidentity of a client. Then, each matcher compares the sub-mitted trait with the template of the claimed identity, andprovides a real-valued matching score: the higher the score,the higher the similarity. We denote the score of the finger-print and the face matcher respectively as xfing and xface.Finally, the matching scores are combined through a properfusion rule to decide whether the claimed identity is theuser’s identity (genuine user) or not (impostor).

As in [1], [2], we consider the widely used likelihoodratio (LLR) score fusion rule [46], and the matching scoresas its features. Denoting as x the feature vector ðxfing; xfaceÞ,the LLR can be written as

LLRðxÞ ¼L if pðxjY ¼ LÞ=pðxjY ¼ MÞ � t;M if pðxjY ¼ LÞ=pðxjY ¼ MÞ < t;

�

(5)

where L and M denote respectively the “genuine” and“impostor” class, and t is a decision threshold set accordingto application requirements. The distributionspðxfing; xfacejY Þ are usually estimated from training data,while t is estimated from a validation set.

1) Attack scenario. Goal. In this case, each malicioususer (impostor) aims at being accepted as a legitimate(genuine) one. This corresponds to a targeted integrityviolation, where the adversary’s goal is to maximize thematching score.

Knowledge. As in [1], [2], we assume that each impostorknows: (k.i) the identity of the targeted client; and (k.ii) thebiometric traits used by the system. No knowledge of (k.iii)the decision function and (k.iv) its parameters is assumed,and (k.v) no feedback is available from the classifier.

Capability. We assume that: (c.i) spoofing attacks affectonly testing data (exploratory attack); (c.ii) they do not affectthe class priors;9 and, according to (k.i), (c.iii) each adver-sary (impostor) controls her testing malicious samples. Welimit the capability of each impostor by assuming that (c.iv)only one specific trait can be spoofed at a time (i.e., only onefeature can be manipulated), and that it is the same for allimpostors. We will thus repeat the evaluation twice, consid-ering fingerprint and face spoofing, separately. Defininghow impostors can manipulate the features (in this case, thematching scores) in a spoofing attack is not straightforward.The standard way of evaluating the security of biometricsystems against spoofing attacks consists of fabricatingfake biometric traits and assessing performance when theyare submitted to the system. However, this is a cumbersometask. Here we follow the approach of [1], [2], in which theimpostor is assumed to fabricate “perfect” fake traits, i.e.,fakes that produce the same matching score of the “live”trait of the targeted client. This allows one to use the avail-able genuine scores to simulate the fake scores, avoiding thefabrication of fake traits. On the other hand, this assumptionmay be too pessimistic. This limitation could be overcome

9. Further, since the LLR considers only the ratio between the class-conditional distributions, changing the class priors is irrelevant.

Fig. 3. AUC10 percent attained on TS as a function of nmax, for the LR (top)and SVM (bottom) classifier, with 1,000 (1K), 2,000 (2K), 10,000 (10K)and 20,000 (20K) features. The AUC10 percent value for nmax ¼ 0, corre-sponding to classical performance evaluation, is also reported in the leg-end between square brackets.

by developing more realistic models of the distribution ofthe fake traits; however, this is a challenging research issueon its own, that is out of the scope of this work, and part ofthe authors’ ongoing work [47], [48].

Attack strategy. The above attack strategy modifies onlythe testing data, and: (a.i) it does not modify the class priors;(a.ii) it does not affect the genuine class, but all impostors;and (a.iii) any impostor spoofs the considered biometric trait(either the face or fingerprint) of the known identity. Accord-ing to the above assumption, as in [1], [2], any spoofing attackis simulated by replacing the corresponding impostor score(either the face or fingerprint) with the score of the targetedgenuine user (chosen at random from the legitimate samplesin the testing set). This can be represented with a functionAðxÞ, that, given an impostor x ¼ ðxfing; xfaceÞ, returns eithera vector ðx0fing; xfaceÞ (for fingerprint spoofing) or ðxfing; x

0faceÞ

(for face spoofing), where x0fing and x0face are the matchingscores of the targeted genuine user.

2) Data model. Since training data is not affected, we setptrðX; Y Þ ¼ pDðX; Y Þ. Then, we set (a.i) ptsðY Þ ¼ pDðY Þ; and(a.ii) ptsðA ¼ TjY ¼ LÞ ¼ 0 and ptsðA ¼ TjY ¼ MÞ ¼ 1.Thus, ptsðXjY ¼ LÞ ¼ pDðXjY ¼ LÞ, while all the malicioussamples in TS have to be manipulated by AðxÞ, according to(a.iii). This amounts to empirically defining ptsðXjY ¼M; A ¼ TÞ. As in the spam filtering case, ptsðXjY ¼ y;A ¼ FÞ, for y ¼ L;M, will be empirically approximated fromthe available data in D.

The definition of the above attack scenario and datamodel is summarized in Table 1 (second column).

3) Construction of TR and TS. We use the NIST BiometricScore Set, Release 1.10 It contains raw similarity scoresobtained on a set of 517 users from two different face match-ers (named ‘G’ and ‘C’), and from one fingerprint matcherusing the left and right index finger. For each user, one gen-uine score and 516 impostor scores are available for eachmatcher and each modality, for a total of 517 genuine and266,772 impostor samples. We use the scores of the ‘G’ facematcher and the ones of the fingerprint matcher for the leftindex finger, and normalize them in ½0; 1� using the min-max technique. The data set D is made up of pairs of faceand fingerprint scores, each belonging to the same user. Wefirst randomly subdivide D into a disjoint training and test-ing set, DTR and DTS, containing respectively 80 and 20 per-cent of the samples. As the attack does not affect thetraining samples, we set TR ¼ DTR. The sets DL;F

TS and DM;FTS

are constructed using DTS, while DL;FTS ¼ ;. The set of attack

samples DM;TTS is obtained by modifying each sample of DM;F

TS

with AðxÞ. We finally set the size of TS as jDTSj, and runAlgorithm 1 to obtain it.

4) Performance evaluation. In biometric authenticationtasks, the performance is usually measured in terms of genu-ine acceptance rate (GAR) and false acceptance rate (FAR),respectively the fraction of genuine and impostor attemptsthat are accepted as genuine by the system. We use here thecomplete ROC curve, which shows the GAR as a function ofthe FAR for all values of the decision threshold t (see Eq. (5)).

To estimate pðXjY ¼ yÞ, for y ¼ L;M, in the LLR scorefusion rule (Eq. (5)), we assume that xfing and xface are

conditionally independent given Y , as usually done in bio-metrics. We thus compute a maximum likelihood estimateof pðxfing; xfacejY Þ from TR using a product of two Gammadistributions, as in [1].

Fig. 4 shows the ROC curve evaluated with the stan-dard approach (without spoofing attacks), and the onescorresponding to the simulated spoofing attacks. TheFAR axis is in logarithmic scale to focus on low FAR val-ues, which are the most relevant ones in security appli-cations. For any operational point on the ROC curve(namely, for any value of the decision threshold t), theeffect of the considered spoofing attack is to increase theFAR, while the GAR does not change. This correspondsto a shift of the ROC curve to the right. Accordingly, theFAR under attack must be compared with the originalone, for the same GAR.

Fig. 4 clearly shows that the FAR of the biometric systemsignificantly increases under the considered attack scenario,especially for fingerprint spoofing. As far as this attack sce-nario is deemed to be realistic, the system designer shouldconclude that the considered system can be evaded byspoofing only one biometric trait, and thus does not exhibita higher security than each of its individual classifiers. Forinstance, in applications requiring relatively high security, areasonable choice may be to chose the operational pointwith FAR ¼ 10�3 and GAR ¼ 0:90, using classical perfor-mance evaluation, i.e., without taking into account spoofingattacks. This means that the probability that the deployedsystem wrongly accepts an impostor as a genuine user (theFAR) is expected to be 0.001. However, under face spoofing,the corresponding FAR increases to 0:10, while it jumps toabout 0:70 under fingerprint spoofing. In other words, animpostor who submits a perfect replica of the face of a clienthas a probability of 10 percent of being accepted as genuine,while this probability is as high as 70 percent, if she is ableto perfectly replicate the fingerprint. This is unacceptablefor security applications, and provides further support tothe conclusions of [1], [2] against the common belief thatmultimodal biometric systems are intrinsically more robustthan unimodal systems.

4.3 Network Intrusion Detection

Intrusion detection systems analyze network traffic to pre-vent and detect malicious activities like intrusion attempts,10. http://www.itl.nist.gov/iad/894.03/biometricscores/.

Fig. 4. ROC curves of the considered multimodal biometric system,under a simulated spoof attack against the fingerprint or the facematcher.

port scans, and denial-of-service attacks.11 When suspectedmalicious traffic is detected, an alarm is raised by the IDSand subsequently handled by the system administrator.Two main kinds of IDSs exist: misuse detectors and anom-aly-based ones. Misuse detectors match the analyzed networktraffic against a database of signatures of known maliciousactivities (e.g., Snort).12 The main drawback is that they arenot able to detect never-before-seen malicious activities, oreven variants of known ones. To overcome this issue, anom-aly-based detectors have been proposed. They build a statis-tical model of the normal traffic using machine learningtechniques, usually one-class classifiers (e.g., PAYL [49]),and raise an alarm when anomalous traffic is detected. Theirtraining set is constructed, and periodically updated to fol-low the changes of normal traffic, by collecting unsuper-vised network traffic during operation, assuming that it isnormal (it can be filtered by a misuse detector, and shouldbe discarded if some system malfunctioning occurs duringits collection). This kind of IDS is vulnerable to causativeattacks, since an attacker may inject carefully designed mali-cious traffic during the collection of training samples toforce the IDS to learn a wrong model of the normal traffic[11], [17], [29], [30], [36].

Here we assume that an anomaly-based IDS is beingdesigned, using a one-class n-SVM classifier with a radialbasis function (RBF) kernel and the feature vector repre-sentation proposed in [49]. Each network packet is con-sidered as an individual sample to be labeled as normal(legitimate) or anomalous (malicious), and is representedas a 256-dimensional feature vector, defined as the histo-gram of byte frequencies in its payload (this is known as“1-gram” representation in the IDS literature). We thenfocus on the model selection stage. In the above setting,it amounts to choosing the values of the n parameter ofthe learning algorithm (which is an upper bound on thefalse positive error rate on training data [50]), and the g

value of the RBF kernel. For the sake of simplicity, weassume that n is set to 0:01 as suggested in [24], so thatonly g has to be chosen.

We show how the IDS designer can select a model (thevalue of g) based also on the evaluation of classifier security.We focus on a causative attack similar to the ones consid-ered in [38], aimed at forcing the learned model of normaltraffic to include samples of intrusions to be attempted dur-ing operation. To this end, the attack samples should becarefully designed such that they include some features ofthe desired intrusive traffic, but do not perform any realintrusion (otherwise, the collected traffic may be discarded,as explained above).

1) Attack scenario. Goal. This attack aims to cause an indis-criminate integrity violation by maximizing the fraction ofmalicious testing samples misclassified as legitimate.

Knowledge. The adversary is assumed to know: (k.ii) thefeature set; and (k.iii) that a one-class classifier is used. No

knowledge of (k.i) the training data and (k.iv) the classifiers’parameters is available to the adversary, as well as (k.v) anyfeedback from the classifier.

Capability. The attack consists of injecting malicious sam-ples into the training set. Accordingly, we assume that: (c.i)the adversary can inject malicious samples into the trainingdata, without manipulating testing data (causative attack);(c.ii) she can modify the class priors by injecting a maximumfraction pmax of malicious samples into the training data;(c.iii) all the injected malicious samples can be manipulated;and (c.iv) the adversary is able to completely control thefeature values of the malicious attack samples. As in [17],[28], [29], [30], we repeat the security evaluation forpmax 2 ½0; 0:5�, since it is unrealistic that the adversary cancontrol the majority of the training data.

Attack strategy. The goal of maximizing the percentage ofmalicious testing samples misclassified as legitimate, forany pmax value, can be attained by manipulating the trainingdata with the following attack strategy: (a.i) the maximumpercentage of malicious samples pmax will be injected intothe training data; (a.ii) all the injected malicious sampleswill be attack samples, while no legitimate samples will beaffected; and (a.iii) the feature values of the attack sampleswill be equal to that of the malicious testing samples. Tounderstand the latter assumption, note that: first, the bestcase for the adversary is when the attack samples exhibitthe same histogram of the payload’s byte values as the mali-cious testing samples, as this intuitively forces the model ofthe normal traffic to be as similar as possible to the distribu-tion of the testing malicious samples; and, second, this doesnot imply that the attack samples perform any intrusion,allowing them to avoid detection. In fact, one may constructan attack packet which exhibit the same feature values of anintrusive network packet (i.e., a malicious testing sample),but looses its intrusive functionality, by simply shuffling itspayload bytes. Accordingly, for our purposes, we directlyset the feature values of the attack samples equal to those ofthe malicious testing samples, without constructing real net-work packets.

2) Data model. Since testing data is not affected by the con-sidered causative attack, we set ptsðX; Y Þ ¼ pDðX; Y Þ. Wethen encode the attack strategy as follows. According to(a.ii), ptrðA ¼ TjY ¼ LÞ ¼ 0 and ptrðA ¼ TjY ¼ MÞ ¼ 1. Theformer assumption implies that ptrðXjY ¼ LÞ ¼ pDðXjY ¼LÞ. Since the training data of one-class classifiers does notcontain any malicious samples (see explanation at the begin-ning of this section), i.e., ptrðA ¼ FjY ¼ MÞ ¼ 0, the latterassumption implies that the class prior ptrðY ¼ MÞ corre-sponds exactly to the fraction of attack samples in the train-ing set. Therefore, assumption (a.i) amounts to settingptrðY ¼ MÞ ¼ pmax. Finally, according to (a.iii), the distribu-tion of attack samples ptrðXjY ¼ M; A ¼ TÞ will be definedas the empirical distribution of the malicious testing samples,namely, DM;T

TR ¼ DM;FTS . The distribution ptrðXjY ¼ L; A ¼

FÞ ¼ pDðXjLÞwill also be empirically defined as the setDL;FTR .

The definition of the attack scenario and data model issummarized in Table 1 (third column).

3) Construction of TR and TS. We use the data set of[24]. It consists of 1,699,822 legitimate samples (networkpackets) collected by a web server during five days in2006, and a publicly available set of 205 malicious

11. The term “attack” is used in this field to denote a maliciousactivity, even when there is no deliberate attempt of misleading an IDS.In adversarial classification, as in this paper, this term is used to specifi-cally denote the attempt of misleading a classifier, instead. To avoidany confusion, in the following we will refrain from using the term“attack” with the former meaning, using paraphrases, instead.

12. http://www.snort.org/.

samples coming from intrusions which exploit the HTTP

protocol [51].13 To construct TR and TS, we take intoaccount the chronological order of network packets as inSection 4.1, and the fact that in this application all themalicious samples in D must be inserted in the testingset [27], [51]. Accordingly, we set DTR as the first 20,000legitimate packets of day one, and DTS as the first 20,000legitimate samples of day two, plus all the malicioussamples. Since this attack does not affect testing samples,we set TS ¼ DTS. We then set DL;F

TR ¼ DTR, andDM;F

TR ¼ DL;TTR ¼ ;. Since the feature vectors of attack sam-

ples are identical to those of the malicious samples, asabove mentioned, we set DM;T

TR ¼ DM;FTS (where the latter

set clearly includes the malicious samples in DTS). Thesize of TR is initially set to 20,000. We then consider dif-ferent attack scenarios by increasing the number ofattack samples in the training data, up to 40,000 samplesin total, that corresponds to pmax ¼ 0:5. For each value ofpmax, TR is obtained by running Algorithm 1 with theproper inputs.

4) Performance evaluation. Classifier performance isassessed using the AUC10 percent measure, for the same rea-sons as in Section 4.1. The performance under attack is eval-uated as a function of pmax, as in [17], [28], which reduces tothe classical performance evaluation when pmax ¼ 0. For thesake of simplicity, we consider only two values of theparameter g, which clearly point out how design choicesbased only on classical performance evaluation methodscan be unsuitable for adversarial environments.

The results are reported in Fig. 5. In the absence ofattacks (pmax ¼ 0), the choice g ¼ 0:5 appears slightly betterthan g ¼ 0:01. Under attack, the performance for g ¼ 0:01remains almost identical as the one without attack, andstarts decreasing very slightly only when the percentage ofattack samples in the training set exceeds 30 percent. On thecontrary, for g ¼ 0:5 the performance suddenly drops aspmax increases, becoming lower than the one for g ¼ 0:01when pmax is as small as about 1 percent. The reason for thisbehavior is the following. The attack samples can be consid-ered outliers with respect to the legitimate training samples,

and, for large g values of the RBF kernel, the SVM discrimi-nant function tends to overfit, forming a “peak” aroundeach individual training sample. Thus, it exhibits relativelyhigh values also in the region of the feature space where theattack samples lie, and this allows many of the correspond-ing testing intrusions. Conversely, this is not true for lowerg values, where the higher spread of the RBF kernel leads toa smoother discriminant function, which exhibits muchlower values for the attack samples.

According to the above results, the choice of g ¼ 0:5, sug-gested by classical performance evaluation for pmax ¼ 0 isclearly unsuitable from the viewpoint of classifier securityunder the considered attack, unless it is deemed unrealisticthat an attacker can inject more than 1 percent attack sam-ples into the training set. To summarize, the designershould select g ¼ 0:01 and trade a small decrease of classifi-cation accuracy in the absence of attacks for a significantsecurity improvement.

5 SECURE DESIGN CYCLE: NEXT STEPS

The classical design cycle of a pattern classifier [9] consistsof: data collection, data pre-processing, feature extractionand selection, model selection (including the choice of thelearning and classification algorithms, and the tuning oftheir parameters), and performance evaluation. We pointedout that this design cycle disregards the threats that mayarise in adversarial settings, and extended the performanceevaluation step to such settings. Revising the remainingsteps under a security viewpoint remains a very interestingissue for future work. Here we briefly outline how this openissue can be addressed.

If the adversary is assumed to have some control overthe data collected for classifier training and parametertuning, a filtering step to detect and remove attack sam-ples should also be performed (see, e.g., the data sanitiza-tion method of [27]).

Feature extraction algorithms should be designed to berobust to sample manipulation. Alternatively, featureswhich are more difficult to be manipulated should be used.For instance, in [52] inexact string matching was proposedto counteract word obfuscation attacks in spam filtering. Inbiometric recognition, it is very common to use additionalinput features to detect the presence (“liveness detection”)of attack samples coming from spoofing attacks, i.e., fakebiometric traits [53]. The adversary could also underminefeature selection, e.g., to force the choice of a set of featuresthat are easier to manipulate, or that are not discriminantenough with respect to future attacks. Therefore, featureselection algorithms should be designed by taking intoaccount not only the discriminant capability, but also therobustness of features to adversarial manipulation.

Model selection is clearly the design step that is moresubject to attacks. Selected algorithms should be robust tocausative and exploratory attacks. In particular, robustlearning algorithms should be adopted, if no data sanitiza-tion can be performed. The use of robust statistics hasalready been proposed to this aim; in particular, to deviselearning algorithms robust to a limited amount of data con-tamination [14], [29], and classification algorithms robust tospecific exploratory attacks [6], [23], [26], [32].13. http://www.i-pi.com/HTTP-attacks-JoCN-2006/.

Fig. 5. AUC10 percent as a function of pmax, for the n-SVMs with RBF ker-nel. The AUC10 percent value for pmax ¼ 0, corresponding to classical per-formance evaluation, is also reported in the legend between squarebrackets. The standard deviation (dashed lines) is reported only forg ¼ 0:5, since it was negligible for g ¼ 0:01.

Finally, a secure system should also guarantee the privacyof its users, against attacks aimed at stealing confidentialinformation [14]. For instance, privacy preserving methodshave been proposed in biometric recognition systems toprotect the users against the so-called hill-climbing attacks,whose goal is to get information about the users’ biometrictraits [41], [53]. Randomization of some classifiers parame-ters has been also proposed to preserve privacy in [14], [54].

6 CONTRIBUTIONS, LIMITATIONS AND OPEN

ISSUES

In this paper we focused on empirical security evaluation ofpattern classifiers that have to be deployed in adversarialenvironments, and proposed how to revise the classical per-formance evaluation design step, which is not suitable forthis purpose.

Our main contribution is a framework for empirical secu-rity evaluation that formalizes and generalizes ideas fromprevious work, and can be applied to different classifiers,learning algorithms, and classification tasks. It is groundedon a formal model of the adversary, and on a model of datadistribution that can represent all the attacks considered inprevious work; provides a systematic method for the gener-ation of training and testing sets that enables security evalu-ation; and can accommodate application-specific techniquesfor attack simulation. This is a clear advancement withrespect to previous work, since without a general frame-work most of the proposed techniques (often tailored to agiven classifier model, attack, and application) could not bedirectly applied to other problems.

An intrinsic limitation of our work is that securityevaluation is carried out empirically, and it is thus data-dependent; on the other hand, model-driven analyses [12],[17], [38] require a full analytical model of the problem andof the adversary’s behavior, that may be very difficult todevelop for real-world applications. Another intrinsic limi-tation is due to fact that our method is not application-spe-cific, and, therefore, provides only high-level guidelines forsimulating attacks. Indeed, detailed guidelines require oneto take into account application-specific constraints andadversary models. Our future work will be devoted todevelop techniques for simulating attacks for differentapplications.

Although the design of secure classifiers is a distinctproblem than security evaluation, our framework could bealso exploited to this end. For instance, simulated attacksamples can be included into the training data to improvesecurity of discriminative classifiers (e.g., SVMs), while theproposed data model can be exploited to design moresecure generative classifiers. We obtained encouraging pre-liminary results on this topic [32].

ACKNOWLEDGMENTS

The authors are grateful to Davide Ariu, Gavin Brown,Pavel Laskov, and Blaine Nelson for discussions and com-ments on an earlier version of this paper. This work waspartly supported by a grant awarded to Battista Biggio byRegione Autonoma della Sardegna, PO Sardegna FSE 2007-2013, L.R. 7/2007 “Promotion of the scientific research and

technological innovation in Sardinia”, by the project CRP-18293 funded by Regione Autonoma della Sardegna, L.R.7/2007, Bando 2009, and by the TABULA RASA project,funded within the seventh Framework Research Pro-gramme of the European Union.

REFERENCES

[1] R.N. Rodrigues, L.L. Ling, and V. Govindaraju, “Robustness ofMultimodal Biometric Fusion Methods against Spoof Attacks,”J. Visual Languages and Computing, vol. 20, no. 3, pp. 169-179, 2009.

[2] P. Johnson, B. Tan, and S. Schuckers, “Multimodal Fusion Vulner-ability to Non-Zero Effort (Spoof) Imposters,” Proc. IEEE Int’lWorkshop Information Forensics and Security, pp. 1-5, 2010.

[3] P. Fogla, M. Sharif, R. Perdisci, O. Kolesnikov, and W. Lee,“Polymorphic Blending Attacks,” Proc. 15th Conf. USENIX Secu-rity Symp., 2006.

[4] G.L. Wittel and S.F. Wu, “On Attacking Statistical Spam Filters,”Proc. First Conf. Email and Anti-Spam, 2004.

[5] D. Lowd and C. Meek, “Good Word Attacks on Statistical SpamFilters,” Proc. Second Conf. Email and Anti-Spam, 2005.

[6] A. Kolcz and C.H. Teo, “Feature Weighting for Improved Classi-fier Robustness,” Proc. Sixth Conf. Email and Anti-Spam, 2009.

[7] D.B. Skillicorn, “Adversarial Knowledge Discovery,” IEEE Intelli-gent Systems, vol. 24, no. 6, Nov./Dec. 2009.

[8] D. Fetterly, “Adversarial Information Retrieval: The Manipulationof Web Content,” ACM Computing Rev., 2007.

[9] R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classification.Wiley-Interscience Publication, 2000.

[10] N. Dalvi, P. Domingos, Mausam, S. Sanghai, and D. Verma,“Adversarial Classification,” Proc. 10th ACM SIGKDD Int’l Conf.Knowledge Discovery and Data Mining, pp. 99-108, 2004.

[11] M. Barreno, B. Nelson, R. Sears, A.D. Joseph, and J.D. Tygar, “CanMachine Learning be Secure?” Proc. ACM Symp. Information, Com-puter and Comm. Security (ASIACCS), pp. 16-25, 2006.

[12] A.A. C�ardenas and J.S. Baras, “Evaluation of Classifiers: PracticalConsiderations for Security Applications,” Proc. AAAI WorkshopEvaluation Methods for Machine Learning, 2006.

[13] P. Laskov and R. Lippmann, “Machine Learning in AdversarialEnvironments,” Machine Learning, vol. 81, pp. 115-119, 2010.

[14] L. Huang, A.D. Joseph, B. Nelson, B. Rubinstein, and J.D. Tygar,“Adversarial Machine Learning,” Proc. Fourth ACM WorkshopArtificial Intelligence and Security, pp. 43-57, 2011.

[15] M. Barreno, B. Nelson, A. Joseph, and J. Tygar, “The Security ofMachine Learning,” Machine Learning, vol. 81, pp. 121-148, 2010.

[16] D. Lowd and C. Meek, “Adversarial Learning,” Proc. 11th ACMSIGKDD Int’l Conf. Knowledge Discovery and Data Mining, pp. 641-647, 2005.

[17] P. Laskov and M. Kloft, “A Framework for Quantitative SecurityAnalysis of Machine Learning,” Proc. Second ACM Workshop Secu-rity and Artificial Intelligence, pp. 1-4, 2009.

[18] NIPS Workshop Machine Learning in Adversarial Environments forComputer Security, http://mls-nips07.first.fraunhofer.de/, 2007.

[19] Dagstuhl Perspectives Workshop Mach. Learning Methods for Com-puter Sec., http://www.dagstuhl.de/12371/, 2012.

[20] A.M. Narasimhamurthy and L.I. Kuncheva, “A Framework forGenerating Data to Simulate Changing Environments,” Proc. 25thConf. Proc. the 25th IASTED Int’l Multi-Conf.: Artificial Intelligenceand Applications, pp. 415-420, 2007.

[21] S. Rizzi, “What-If Analysis,” Encyclopedia of Database Systems,pp. 3525-3529, Springer, 2009.

[22] J. Newsome, B. Karp, and D. Song, “Paragraph: Thwarting Signa-ture Learning by Training Maliciously,” Proc. Ninth Int’l Conf.Recent Advances in Intrusion Detection, pp. 81-105, 2006.

[23] A. Globerson and S.T. Roweis, “Nightmare at Test Time: RobustLearning by Feature Deletion,” Proc. 23rd Int’l Conf. Machine Learn-ing, pp. 353-360, 2006.

[24] R. Perdisci, G. Gu, and W. Lee, “Using an Ensemble of One-ClassSVM Classifiers to Harden Payload-Based Anomaly DetectionSystems,” Proc. Int’l Conf. Data Mining, pp. 488-498, 2006.

[25] S.P. Chung and A.K. Mok, “Advanced Allergy attacks: Does aCorpus Really Help,” Proc. 10th Int’l Conf. Recent Advances in Intru-sion Detection (RAID ’07), pp. 236-255, 2007.

[26] Z. Jorgensen, Y. Zhou, and M. Inge, “A Multiple Instance Learn-ing Strategy for Combating Good Word Attacks on Spam Filters,”J. Machine Learning Research, vol. 9, pp. 1115-1146, 2008.

[27] G.F. Cretu, A. Stavrou, M.E. Locasto, S.J. Stolfo, and A.D. Keromy-tis, “Casting out Demons: Sanitizing Training Data for AnomalySensors,” Proc. IEEE Symp. Security and Privacy, pp. 81-95, 2008.

[28] B. Nelson, M. Barreno, F.J. Chi, A.D. Joseph, B.I.P. Rubinstein, U.Saini, C. Sutton, J.D. Tygar, and K. Xia, “Exploiting MachineLearning to Subvert Your Spam Filter,” Proc. First Workshop Large-Scale Exploits and Emergent Threats, pp. 1-9, 2008.

[29] B.I. Rubinstein, B. Nelson, L. Huang, A.D. Joseph, S.-h. Lau, S.Rao, N. Taft, and J.D. Tygar, “Antidote: Understanding andDefending against Poisoning of Anomaly Detectors,” Proc. NinthACM SIGCOMM Internet Measurement Conf. (IMC ’09), pp. 1-14,2009.

[30] M. Kloft and P. Laskov, “Online Anomaly Detection under Adver-sarial Impact,” Proc. 13th Int’l Conf. Artificial Intelligence and Statis-tics, pp. 405-412, 2010.

[31] O. Dekel, O. Shamir, and L. Xiao, “Learning to Classify with Miss-ing and Corrupted Features,” Machine Learning, vol. 81, pp. 149-178, 2010.

[32] B. Biggio, G. Fumera, and F. Roli, “Design of Robust Classifiers forAdversarial Environments,” Proc. IEEE Int’l Conf. Systems, Man,and Cybernetics, pp. 977-982, 2011.

[33] B. Biggio, G. Fumera, and F. Roli, “Multiple Classifier Systems forRobust Classifier Design in Adversarial Environments,” Int’l J.Machine Learning and Cybernetics, vol. 1, no. 1, pp. 27-41, 2010.

[34] B. Biggio, I. Corona, G. Fumera, G. Giacinto, and F. Roli, “BaggingClassifiers for Fighting Poisoning Attacks in Adversarial Environ-ments,” Proc. 10th Int’l Workshop Multiple Classifier Systems,pp. 350-359, 2011.

[35] B. Biggio, G. Fumera, F. Roli, and and L. Didaci, “PoisoningAdaptive Biometric Systems,” Proc. Joint IAPR Int’l Conf. Struc-tural, Syntactic, and Statistical Pattern Recognition, pp. 417-425, 2012.

[36] B. Biggio, B. Nelson, and P. Laskov, “Poisoning Attacks againstSupport Vector Machines,” Proc. 29th Int’l Conf. Machine Learning,2012.

[37] M. Kearns and M. Li, “Learning in the Presence of MaliciousErrors,” SIAM J. Computing, vol. 22, no. 4, pp. 807-837, 1993.

[38] A.A. C�ardenas, J.S. Baras, and K. Seamon, “A Framework for theEvaluation of Intrusion Detection Systems,” Proc. IEEE Symp.Security and Privacy, pp. 63-77, 2006.

[39] B. Biggio, G. Fumera, and F. Roli, “Multiple Classifier Systems forAdversarial Classification Tasks,” Proc. Eighth Int’l Workshop Mul-tiple Classifier Systems, pp. 132-141, 2009.

[40] M. Br€uckner, C. Kanzow, and T. Scheffer, “Static PredictionGames for Adversarial Learning Problems,” J. Machine LearningResearch, vol. 13, pp. 2617-2654, 2012.

[41] A. Adler, “Vulnerabilities in Biometric Encryption Systems,” Proc.Fifth Int’l Conf. Audio- and Video-Based Biometric Person Authentica-tion, pp. 1100-1109, 2005.

[42] B. Efron and R.J. Tibshirani, An Introduction to the Bootstrap. Chap-man & Hall, 1993.

[43] H. Drucker, D. Wu, and V.N. Vapnik, “Support Vector Machinesfor Spam Categorization,” IEEE Trans. Neural Networks, vol. 10,no. 5, pp. 1048-1054, Sept. 1999.

[44] F. Sebastiani, “Machine Learning in Automated Text Catego-rization,” ACM Computing Surveys, vol. 34, pp. 1-47, 2002.

[45] C.-C. Chang, C.-J. Lin, “LibSVM: A Library for Support VectorMachines,” http://www.csie.ntu.edu.tw/~cjlin/libsvm/, 2001.

[46] K. Nandakumar, Y. Chen, S.C. Dass, and A. Jain, “LikelihoodRatio-Based Biometric Score Fusion,” IEEE Trans. Pattern Analysisand Machine Intelligence, vol. 30, no. 2, pp. 342-347, Feb. 2008.

[47] B. Biggio, Z. Akhtar, G. Fumera, G. Marcialis, and F. Roli,“Robustness of Multi-Modal Biometric Verification Systems underRealistic Spoofing Attacks,” Proc. Int’l Joint Conf. Biometrics, pp. 1-6, 2011.

[48] B. Biggio, Z. Akhtar, G. Fumera, G.L. Marcialis, and F. Roli,“Security Evaluation of Biometric Authentication Systems underReal Spoofing Attacks,” IET Biometrics, vol. 1, no. 1, pp. 11-24,2012.

[49] K. Wang and S.J. Stolfo, “Anomalous Payload-Based NetworkIntrusion Detection,” Proc. Seventh Symp. Recent Advances in Intru-sion Detection (RAID), pp. 203-222, 2004.

[50] B. Sch€olkopf, A.J. Smola, R.C. Williamson, and P.L. Bartlett, “NewSupport Vector Algorithms,” Neural Computation, vol. 12, no. 5,pp. 1207-1245, 2000.

[51] K. Ingham and H. Inoue, “Comparing Anomaly Detection Tech-niques for http,” Proc. 10th Int’l Conf. Recent Advances in IntrusionDetection, pp. 42-62, 2007.

[52] D. Sculley, G. Wachman, and C.E. Brodley, “Spam Filtering UsingInexact String Matching in Explicit Feature Space with on-LineLinear Classifiers,” Proc. 15th Text Retrieval Conf., 2006.

[53] Encyclopedia of Biometrics, S.Z. Li, and A.K. Jain, eds., Springer US,2009.

[54] B. Biggio, G. Fumera, and F. Roli, “Adversarial Pattern Classifica-tion Using Multiple Classifiers and Randomisation,” Proc. JointIAPR Int’l Workshop Structural, Syntactic, and Statistical Pattern Rec-ognition, pp. 500-509, 2008.

Battista Biggio received the MSc degree inelectronic engineering, with honors, and the PhDdegree in electronic engineering and computerscience, respectively, in 2006 and 2010, from theUniversity of Cagliari, Italy. Since 2007, he hasbeen working for the Department of Electricaland Electronic Engineering at the same Univer-sity, where he holds now a postdoctoral position.From May 12th, 2011 to November 12th, 2011,he visited the University of T€ubingen, Germany,and worked on the security of machine learning

algorithms to contamination of training data. His research interests cur-rently include: secure / robust machine learning and pattern recognitionmethods, multiple classifier systems, kernel methods, biometric authen-tication, spam filtering, and computer security. He serves as a reviewerfor several international conferences and journals, including PatternRecognition and Pattern Recognition Letters. He is a member of theIEEE Computer Society, and of the Italian Group of Italian Researchersin Pattern Recognition (GIRPR), affiliated to the International Associa-tion for Pattern Recognition.

Giorgio Fumera received the MSc degree inelectronic engineering, with honors, and thePhD degree in electronic engineering and com-puter science, respectively, in 1997 and 2002,from the University of Cagliari, Italy. Since Feb-ruary 2010, he has been an associate professorof computer engineering in the Department ofElectrical and Electronic Engineering at thesame University. His research interests arerelated to methodologies and applications ofstatistical pattern recognition, and include multi-

ple classifier systems, classification with the reject option, adversarialclassification, and document categorization. On these topics he pub-lished more than 60 papers in international journal and conferences.He acts as reviewer for the main international journals in this field,including IEEE Transactions on Pattern Analysis and Machine Intelli-gence, Journal of Machine Learning Research, and Pattern Recogni-tion. He is a member of the IEEE Computer Society, of the ItalianAssociation for Artificial Intelligence (AI*IA), and of the Italian Groupof Italian Researchers in Pattern Recognition (GIRPR), affiliated tothe International Association for Pattern Recognition.

Fabio Roli received the MSc degree, with hon-ors, and the PhD degree in electronic engineer-ing from the University of Genoa, Italy. He was amember of the research group on Image Proc-essing and Understanding of the University ofGenoa, Italy, from 1988 to 1994. He was anadjunct professor at the University of Trento,Italy, in 1993 and 1994. In 1995, he joined theDepartment of Electrical and Electronic Engi-neering of the University of Cagliari, Italy, wherehe is currently a professor of computer engineer-

ing and the head of the research group on pattern recognition and appli-cations. His research activity is focused on the design of patternrecognition systems and their applications to biometric personal identifi-cation, multimedia text categorization, and computer security. On thesetopics, he has published more than 200 papers at conferences and onjournals. He was a very active organizer of international conferencesand workshops, and established the popular workshop series on multipleclassifier systems. He is a member of the governing boards of the Inter-national Association for Pattern Recognition and of the IEEE Systems,Man and Cybernetics Society. He is fellow of the IEEE, and a fellow ofthe International Association for Pattern Recognition.