Evolutionary Fuzzy K-Nearest Neighbors Algorithm using ...chiclana/publications/INS-D-14-1388R1.pdf · In nearest neighbor classi cation, fuzzy sets can be used to model the degree

Evolutionary Fuzzy K-Nearest Neighbors Algorithm using

Interval-Valued Fuzzy Sets

Joaquın Derraca,1, Francisco Chiclanab, Salvador Garcıac, Francisco Herrerad,e

aAffectv: Affectv Limited, 77 Oxford Street, London, W1D 2ES, United KingdombCentre for Computational Intelligence (CCI), Faculty of Technology, De Montfort University, Leicester

LE1 9BH, United KingdomcDepartment of Computer Science. University of Jaen, 23071 Jaen, Spain

dDept. of Computer Science and Artificial Intelligence. University of Granada, 18071 Granada, SpaineFaculty of Computing and Information Technology - North Jeddah, King Abdulaziz University, 21589,

Jeddah, Saudi Arabia

Abstract

One of the most known and effective methods in supervised classification is the K-NearestNeighbors classifier. Several approaches have been proposed to enhance its precision, withthe Fuzzy K-Nearest Neighbors (Fuzzy-kNN) classifier being among the most successfulones. However, despite its good behavior, Fuzzy-kNN lacks of a method for properly definingseveral mechanisms regarding the representation of the relationship between the instancesand the classes of the classification problems. Such a method would be very desirable, sinceit would potentially lead to an improvement in the precision of the classifier.

In this work we present a new approach, Evolutionary Fuzzy K-Nearest Neighbors clas-sifier using Interval-Valued Fuzzy Sets (EF-kNN-IVFS), incorporating interval-valued fuzzysets for computing the memberships of training instances in Fuzzy-kNN. It is based on therepresentation of multiple choices of two key parameters of Fuzzy-kNN: One is applied inthe definition of the membership function, and the other is used in the computation ofthe voting rule. Besides, evolutionary search techniques are incorporated to the model asa self-optimization procedure for setting up these parameters. An experimental study hasbeen carried out to assess the capabilities of our approach. The study has been validated byusing nonparametric statistical tests, and remarks the strong performance of EF-kNN-IVFScompared with several state of the art techniques in fuzzy nearest neighbor classification.

Keywords: Fuzzy Nearest Neighbor, Interval-Valued Fuzzy Sets, Evolutionary Algorithms,Supervised Learning, Classification.

∗Corresponding author. Tel.: +34 958 240598; fax: +34 958 243317.Email addresses: [email protected] (Joaquın Derrac), [email protected] (Francisco

Chiclana), [email protected] (Salvador Garcıa), [email protected] (Francisco Herrera)

Preprint submitted to Information Sciences November 13, 2014

1. Introduction

The k-Nearest Neighbors classifier (kNN) [15] is one of the most popular supervisedlearning methods. It is a nonparametric method which does not rely on building a modelduring the training phase, and whose classification rule is based on a given similarity functionbetween the training instances and the test instance to be classified. Since its definition,kNN has become one of most relevant algorithms in data mining [41], and it is an integralpart of many applications of machine learning in various domains [34, 38].

In nearest neighbor classification, fuzzy sets can be used to model the degree of mem-bership of each instance to the classes of the problem. This approach, known as the FuzzyK-Nearest Neighbor (Fuzzy-kNN) classifier [30], has been shown to be an effective improve-ment of kNN.

This fuzzy approach overcomes a drawback of the kNN classifier, in which equal im-portance is given to every instance in the decision rule, regardless of its typicalness as aclass prototype and its distance to the pattern to be classified. Fuzzy memberships enableFuzzy-kNN to achieve higher accuracy rates in most classification problems. This is alsothe reason why it has been the preferred choice in several applications in medicine [8, 11],economy [10], bioinformatics [29], industry [32] and many other fields.

The definition of fuzzy memberships is a fundamental issue in Fuzzy-kNN. Although theycan be set through expert knowledge, or by analyzing local data around each instance (as in[30] or [43]), there may be still a lack of knowledge associated with the assignation of a singlevalue to the membership. This is caused by the necessity of fixing in advance two parameters:kInit in the definition of the initial membership values and m in the computation of thevotes of the neighbors.

To overcome this difficulty, interval valued fuzzy sets (IVFSs) [4, 25], a particular case oftype-2 fuzzy sets [33], may be used. IVFSs allow membership values to be defined by using alower and an upper bound. The interval based definition includes not only a greater degreeof flexibility than just using a single value, but also enables us to measure the degree ofignorance with the length of the interval [7, 19]. Following this approach, IVFSs have beensuccessfully applied in the development of fuzzy systems for classification [35, 36, 37]. Inthe case of nearest neighbor classification, this enables the representation of the uncertaintyassociated with the true class (or classes) to which every instance belongs, in the context ofmost standard, supervised classification problems.

The optimization capabilities of evolutionary algorithms can also help to overcome thisissue. In recent years, they have become a very useful tool in the design of fuzzy learn-ing systems. For example, Genetic Fuzzy Systems [14, 13] show how the incorporation ofevolutionary algorithms allows to enhance the performance of the learning model through pa-rameter adjustment. Nearest neighbor classifiers’ performance is also prone to be improvedby the use of Evolutionary algorithms [9, 17].

Considering the aforementioned issue, in this paper we propose an Evolutionary FuzzyK-Nearest Neighbors classifier using Interval-Valued Fuzzy Sets (EF-kNN-IVFS). On theone hand, it tackles the problem of setting up the parameters via the implementation ofinterval values to represent both the membership of each training instance to the classes

2

and the votes cast by each neighbor in the decision rule. The introduction of intervals inthis approach allows us to consider different values for the kInit andm parameters, obtainingas a result different degrees of membership per each training instance. On the other hand,the implementation of evolutionary in the model would enable us to optimize the selection ofboth parameters, thus improving the accuracy of the whole classifier algorithm. Specifically,it is proposed to use evolutionary algorithms to develop an automatic method, driven bythe CHC evolutionary algorithm [20], for optimizing the procedure to build the intervals inthe interval-valued model and, following a wrapper based approach, to adapt the intervalsto the specific chosen data set.

The methodology developed in [18] for the field of fuzzy nearest neighbor classificationis followed to carry out an experimental study to compare the EF-kNN-IVFS and variousadvanced fuzzy nearest neighbor classifiers. In this study, the classification accuracy istested over several well-known classification problems. The results are contrasted usingnonparametric statistical procedures, validating the conclusions drawn from them.

The rest of the paper is organized as follows: Section 2 describes the kNN and Fuzzy-kNNclassifiers, highlighting the enhancements to the former introduced by the latter. Section 3presents the EF-kNN-IVFS model, as a natural extension of Fuzzy-kNN. Section 4 is devotedto the experimental study and the analysis of its results. Finally, conclusions are drawn inSection 5.

2. kNN and Fuzzy-kNN classifiers

The kNN and Fuzzy-kNN classifiers require to measure the similarity of a new queryinstance (the new instance to be classified) to the instances stored in the training set. Inthe next step, a set of k nearest neighbors is found. Every neighbor casts a vote on the classto which the query instance should be assigned. Finally, a class is assigned to the queryinstance by combining these votes.

The above procedure can be formally described as follows: Let X be a training set,composed of N instances X = {x0, x1, . . . , xN} which belong to C classes. Each instancexi = (xi0, x

i1, . . . , x

iM , x

iω) is characterized by M input attributes and one output attribute

ω (ω ∈ C). For a new query instance Q, a nearest neighbor classifier finds its k nearestneighbors in X, using a particular similarity function. Next, the class of Q is predicted asthe aggregation of the class attributes ω of the k nearest neighbors.

Initially, training instances of kNN are labeled using a hard scheme: The membership Uof an instance x to each class of C is given by an array of values in {0, 1}, where Uω(x) = 1and Uc(x) = 0, c ∈ C, c 6= ω. In this scheme, each instance belongs completely to one classand does not belong to any of the rest.

In the case of Fuzzy-kNN [30], the above scheme is extended using a continuous range ofmembership: Memberships are quantified in [0, 1], and obtained using the following mem-bership function

Uc(x) =

{0.51 + (nnc/kInit) ∗ 0.49 if c = ω

(nnc/kInit) ∗ 0.49 otherwise(1)

3

where nnc are the number of instances belonging to class c found among the kInit 1 neighborsof x.

This fuzzy scheme causes instances close to the center of the classes to keep the originalcrisp memberships in {0, 1}, but instances close to the boundaries will spread half of theirmembership between the neighbors’ classes.

Once a query instance Q has been presented, its k nearest neighbors are searched inthe training set. Although many different similarity functions can be considered for thistask, the preferred choice in nearest neighbor classification is to define it via the Euclideandistance, which should suit most classification problems if the training data is normalizedin the domain [0, 1]. Throughout the rest of the paper we will follow this methodology: theEuclidean distance is used and the attributes are normalized.

Once the k nearest neighbors have been determined, the final output of the classifier isobtained by aggregating the votes cast by its neighbors. In the case of kNN, the votes areobtained by simply adding the memberships of the k neighbors. In the case of Fuzzy-kNN,the Euclidean norm and the memberships are weighted to produce a final vote for each classand neighbor using Equation (2):

V (kj, c) =Uc(kj) · 1/(‖Q− kj‖)2/(m−1)∑k

i=1 1/(‖Q− ki‖)2/(m−1)(2)

where kj is the j-th nearest neighbor (kj ∈ k) and m,m > 1 is a parameter used to intensifythe distances between the query Q and the training data set elements (generally m = 2).The votes of each neighbor are finally added to obtain the final classification, as in the caseof kNN.

Thus, both classifiers obtain their final output applying the majority simple rule to theclasses of the k nearest neighbors. However, in the case of Fuzzy-kNN, the use of the softlabeling scheme and the weighted votes allows it to achieve a more precise classification,particularly for instances located next to the decision boundaries, whose crisp classificationwould otherwise be unclear.

3. EF-kNN-IVFS: Evolutionary Fuzzy K-Nearest Neighbors classifier using Interval-Valued Fuzzy Sets

EF-kNN-IVFS is proposed to tackle the problem of membership assignation throughthe introduction of IVFS and evolutionary algorithms. As a consequence, the membershipvalues of every instance in the training set are represented as an array of intervals, depictinga more flexible representation of the typicalness of the instances in every class of the problem.Intervals are also considered in the computation of the votes cast by each of the k nearestneighbors in the decision rule. Using this approach we aim at reducing the sensitivity ofthe original Fuzzy-kNN classifier to the kInit and m parameters, removing the necessity ofcomputing explicitly an ad-hoc configuration for each different data set tackled.

1kInit is usually set to an integer value between [3, 9]

4

After the definition of the core model of EF-kNN-IVFS, evolutionary algorithms areintroduced for optimizing the representation of both the membership values and the votes.These representations are upgraded through a second redefinition of the way in which thekInit and m parameters are interpreted. In our implementation, we have chosen the CHCevolutionary model [20] to conduct the search.

The description of our proposals is organized as follows:

• Firstly, we present the mechanism developed to perform the computation of interval-valued memberships to the classes (Subsection 3.1).

• Secondly, we describe the voting procedure based on intervals (Subsection 3.2).

• Thirdly, we show how the votes can be combined in order to obtain the final classifi-cation (Subsection 3.3).

• Finally, we detail how the evolutionary search is conducted, describing how the kInitand m parameters are adjusted (Subsection 3.4).

3.1. Computation of interval-valued memberships to the classes

In Fuzzy-kNN, the definition of the memberships of the training instances is governedby Equation (1). It is designed so that the class to which an instance originally belongsobtains more than half (0.51) of the total membership, whereas the rest is shared amongthe rest of the classes of the problem. By searching for the kInit nearest instances, localinformation about the relative neighborhood of the instance is considered. Therefore, thisset up incorporates both expert knowledge (the ω classes already assigned in the originaldata) and structural knowledge, thereby obtaining a more accurate representation of thetrue nature of the instance than by use of the kNN classifier [30].

However, a drawback of this approach is that kInit must be fixed in advance. Somerules of thumb may be considered when aiming for a proper set up, such as not setting itto an extremely low value - with kInit = 1 or kInit = 2 very few neighbors are included,and hence most of the local structural information about the data is lost - or not setting itto a very high value - which would make memberships approximately equal to the globaldistribution of classes in the training data, and thus discarding again the local information.Beyond this, any fixed value of kInit could potentially be selected.

We argue that the use of IVFSs to represent membership of classes could provide analternative and efficient solution to the above drawback, and therefore make the fixing of aspecific value to kInit superfluous. Indeed, the use of interval-values for membership couldaccommodate the simultaneous use of different values of kInit. That is, Equation (1) canbe parametrized with kInit

Uc(x, kInit) =

{0.51 + (nnc/kInit) ∗ 0.49 if c = ω

(nnc/kInit) ∗ 0.49 otherwise(3)

and then the membership of a training instance x to a class c can be represented as aninterval

5

Uc(x) = [Uc−left(x), Uc−right(x)] (4)

Uc−left(x) = min [Uc(x, kInit)]

Uc−right(x) = max [Uc(x, kInit)] (5)

with kInit belonging to a particular set of values derived from the application of the con-siderations mentioned before.

Following this scheme, a more flexible and accurate representation of the training in-stances is obtained:

• Instances located at the center of their respective classes, surrounded only by instancesof the same class, will maintain full membership to it ([1.0, 1.0]) and null membershipto the rest of the classes ([0.0, 0.0]). This is equivalent to Fuzzy-kNN and kNN.

• Instances located near the boundaries between classes, surrounded by instances of thesame class, but also by some instances of other classes, will have their membershipsmodified as follows:

– The lower value of the membership to ω, Uc−left(x), may be regarded as a measureof how many neighboring instances with a class different to ω there are, and oftheir relevance. The higher the number of these neighboring instances to thetraining instance they will be, the closer to 0.51 this lower value will be.

– The upper value of the membership to ω, Uc−right(x), is a direct measure of howfar away the first neighbor not belonging to ω is. It will be 1.0 if it is not amongthe first nearest neighbors (in accordance with the set up for kInit chosen), andslightly lower if it is, with the specific value again dependent on the number andposition of the neighboring instances not belonging to ω.

– The lower value of the membership to the rest of classes will be 0.0, unless one ofthe first nearest neighbors belongs to that class. The upper value can be regardedas a relative measure of the presence of this class among the neighborhood of thetraining instance, never greater than 0.49.

• Instances badly identified (possibly noise), surrounded only by instances of otherclasses, will get only half membership to ω ([0.51, 0.51]) whereas the membership tothe rest of the classes will be a representation of the true nature of the instances.

3.2. Interval-valued voting procedure

The votes cast by each neighbor in the computation of the decision rule (Equation 2)can also be represented by intervals. In this expression, the parameter m can be used tovary the influence of the neighbors, depending on the specific value chosen.

6

If m = 2, the vote of each neighbor is weighted by the reciprocal of the squared Man-hattan distance. As m increases, distances between the different neighbors will be evenlyweighted, and thus the relative distances will have less effect on the determination of thevotes (with m = 3 the weight becomes the reciprocal of the Euclidean distance). Similarly,if m is decreased, the relative distances will have a greater effect, reducing the contributionof the furthest instances (as m approaches 1).

Although the choice recommended in [30] was to simply let m = 2, it is possible toconsider this parameter in a more flexible way, by introducing intervals. This allows a rangeof possible values of m to be considered instead of a single one, resulting in a more generalvoting mechanism.

To represent this, Equation 2 becomes:

V (kj, c) = Uc(kj) ·D(kj) (6)

where

D(kj) = [min(D(kj,ma), D(kj,mb)),

max(D(kj,ma), D(kj,mb))] (7)

D(kj,m) =1/(‖Q− kj‖)2/(m−1)∑ki=1 1/(‖Q− ki‖)2/(m−1)

(8)

and ma,mb are the minimum and maximum values chosen for the parameter m. Note thatsince the elements of Equation (6) are intervals, their product must be computed as follows[26]:

[a1, a2] ∗ [b1, b2] = [min(Ij),max(Ij)]

Ij = {a1 · b1, a1 · b2, a2 · b1, a2 · b2} (9)

3.3. Combination of votes

After the votes have been computed, the final classification is obtained as the class withthe maximum vote overall. In the case of EF-kNN-IVFS, the votes for every class arecomputed as

V (c) =k∑j=1

V (kj, c) (10)

where the addition of two intervals is obtained as follows:

[a1, a2] + [b1, b2] = [a1 + b1, a2 + b2] (11)

7

Figure 1: A valid configuration of EF-kNN-IVFS. The values of kInit chosen are 2,3,4 and 8 (from theinterval (1, 9)) and the values of ma and mb are 1.48 and 2.07, respectively

After the votes for every class have been added, the final classification is obtained by applyingan order operator to the intervals obtained for each class. Specifically, a lexicographic orderwith respect to the lower bound [5, 6] has been chosen (further details are provided in theexperimental study, in Section 4.2). Upon the application of this operator, an order can beestablished between the classes, choosing the first one as the output of the classifier.

3.4. Evolutionary optimization of the kInit and m parameters

As is detailed in Sections 3.1 and 3.2, the performance of the classifier is dependent onthe selection of values considered for the kInit, ma and mb parameters. Although propervalues may be fixed experimentally, in this subsection we propose the use of evolutionaryalgorithms for optimizing this selection.

For parameters ma and mb, the optimization procedure needs to find two real valueswithin a reasonable range (in accordance with the definition of Equation 8). However, theselection of a set of values for kInit may be more sophisticated.

Instead of just fixing this selection to an interval (for example, {3, 4, 5, 6, 7, 8, 9}), it ispossible to define an optimization procedure for choosing specific, non-correlative values.

Figure 1 shows an example of a valid configuration for EF-kNN-IVFS. It is is composedof a binary array of Sn digits, in which a value Sn = 1 shows that n is chosen as a value forkInit, while a value Sn = 0 shows that n is not chosen, and two real values, correspondingto ma and mb.

In the example illustrated in Figure 1, the interval [1.48,2.07] has been chosen for them parameter. The values 2,3,4 and 8 have been chosen for the kInit parameter. Note thatthis set-up is similar to the configuration {2, 3, 4, 5, 6, 7, 8}, but it will not produce the samememberships, as the values 5,6 and 7 are not considered. Thus, this configuration could beuseful in such cases where the memberships should be constrained to the 4 nearest neighborsof each training instance, but then extended by considering an instance at a greater distance(the eighth neighbor).

Nevertheless, with this representation, an optimization algorithm may be applied in orderto obtain the best possible configuration for the specific data analyzed. In this sense, geneticalgorithms are a suitable option given its well-known capabilities for optimization problems.

We have chosen the CHC evolutionary algorithm because of its outstanding capabilitiesfor binary optimization [20] and real-coded optimization [21]. It is a robust generational

8

genetic algorithm which involves the combination of an advanced selection strategy with avery highly selective pressure, and several components inducing a strong diversity.

The CHC algorithm is configured as follows:

• Representation of solutions: Binary chromosomes of Sn bits (representing the valueschosen for kInit) and 2 real-coded values (representing the ma and mb values). Figure1 is, in fact, a valid chromosome of the algorithm.

• Initialization: All chromosomes are initialized randomly. Real values are initializedinto the interval ]1, 4] (m cannot take a value of 1 due to the definition of Equation(8), and should never be set to a very high value).

• Crossover operator: HUX and BLX-0.5. The HUX operator (the classic crossoveroperator of CHC, which interchanges half of the non-matching genes of each parent)is used for the binary part of the solutions, whereas the BLX-0.5 operator [21] is usedfor crossing the real coded values.

• Fitness function: The core model is used as a wrapper. To evaluate a solution, EF-kNN-IVFS is configured with the parameters represented by the current solution, and itsaccuracy over the training data (using a leave-one-out validation scheme) is measured.This accuracy is considered to be the fitness of the solution.

The rest of the elements follow the same configuration as in the original definition ofCHC.

In summary, the application of CHC to optimize the configuration of the core modelenables us to obtain a suitable configuration, adapted to the training data available, withoutthe necessity of a specific set-up provided by the practitioner. This increases the adaptabilityof the whole model, EF-kNN-IVFS, for general supervised learning problems, providing thatthe training data available is representative of the problem considered.

4. Experimental study

An experimental study has been carried out to test the performance of EF-kNN-IVFS.The experiments will involve several well-known classification problems and various stateof the art algorithms in fuzzy nearest neighbor classification, chosen according to the revi-sion presented in [18]. Section 4.1 describes the experimental framework in which all theexperiments have been carried out. Section 4.2 provides a description on the study per-formed for choosing the right order operator for comparing the intervals that represent theoutput of EF-kNN-IVFS. Section 4.3 shows the results obtained in the main experiments.Finally, Section 4.4 shows a study on the improvement obtained through the incorporationof evolutionary algorithms to the core model of EF-kNN-IVFS.

9

Table 1: Data sets considered in the study

Data set #Ins. #At. #Cl. Data set #Ins. #At. #Cl.

Appendicitis 106 7 2 Penbased 10992 16 10Balance 625 4 3 Phoneme 5404 5 2Banana 5300 2 2 Pima 768 8 2Bands 539 19 2 Ring 7400 20 2Bupa 345 6 2 Satimage 6435 36 7Cleveland 297 13 5 Segment 2310 19 7Dermatology 358 34 6 Sonar 208 60 2Ecoli 336 7 8 Spambase 4597 57 2Glass 214 9 7 Spectfheart 267 44 2Haberman 306 3 2 Tae 151 5 3Hayes-roth 160 4 3 Texture 5500 40 11Heart 270 13 2 Thyroid 7200 21 3Hepatitis 80 19 2 Titanic 2201 3 2Ionosphere 351 33 2 Twonorm 7400 20 2Iris 150 4 3 Vehicle 946 18 4Led7Digit 500 7 10 Vowel 990 13 11Mammographic 830 5 2 Wdbc 569 30 2Marketing 6876 13 9 Wine 178 13 3Monk-2 432 6 2 Winequality-red 1599 11 11Movement 360 90 15 Winequality-white 4898 11 11New Thyroid 215 5 3 Wisconsin 683 9 2Page-blocks 5472 10 5 Yeast 1484 8 10

4.1. Experimental framework

The experiments has been conducted over 44 classification data sets, whose partitions areavailable at the KEEL-Dataset repository2 [2, 1]. Table 1 summarizes their characteristics:number of instances (#Ins.), number of attributes (#At.) and number of classes (#Cl.).None of the data sets includes missing values, and no nominal (discrete) attributes have beenconsidered. All Attribute values have been normalized at [0, 1] and a 10-folds cross validationprocedure has been followed throughout the experiments (the cross validation procedure hasbeen repeated 5 times and the results have been averaged for stochastic algorithms).

Besides EF-kNN-IVFS, we have considered 8 comparison algorithms:

• The kNN classifier [15].

• The Fuzzy-kNN classifier [30].

• D-SKNN [16] is a nearest neighbor classifier based on the Dempster-Shafer theory,incorporating mechanisms to manage uncertainty and rejection of unclear instances.It estimates expected lower and higher classification costs of the instances accordingwith their k nearest neighbors in the training set, assigning the instances to the classwith minimum cost (maximum belief).

2http://www.keel.es/datasets.php

10

• IF-KNN [31] is a method which incorporates intuitionistic fuzzy sets into the votingrule. Votes are weighted using the concepts of membership and nonmembership to acertain class: If the membership of a voting instance is above a lower threshold andthe nonmembership below a higher threshold , the vote is considered as positive. Else,the vote is considered as negative.

• FENN [42] is the Fuzzy version of on Wilson’s editing rule for k-NN, ENN [40]: Allinstances in the training set are checked and those whose classification by the Fuzzy-kNN classifier does not agree with its original class are removed.

• IT2FKNN [12] is a fuzzy nearest neighbor classifier based on interval type-2 fuzzy sets.They are introduced to represent the memberships of the training instances. In thisway, several values for the parameter kInit are considered, thus including flexibilityin the memberships’ representation. Type-2 fuzzy sets are built considering all thedifferent memberships computed, and then a type reduction operation is performedto obtain a combined value, representative of all the choices considered initially. Thealgorithm is similar to Fuzzy-kNN in respect to the rest of the phases.

• GAFuzzy-kNN [28] is the only evolutionary approach available for fuzzy nearest neigh-bor classification, to the best of our knowledge. It is a wrapper-based Fuzzy-kNNclassifier, which works by optimizing the m and kInit parameters using a binary ge-netic algorithm. The fitness function described uses the accuracy of an underlyingFuzzy-KNN classifier as the fitness value.

• PFKNN [3] is a fuzzy prototype reduction method which starts by building a set of pro-totypes representing the border points of different clusters in the data. Then, it addsto this reference set those instances which could be misclassified. The algorithm alsoincludes a third pruning phase in which non relevant prototypes for the classificationphase are discarded.

Table 2 shows the parameters configuration selected for each algorithm (we have followedthe same configuration as in [18]). The parameter k is chosen for each algorithm in the{3,5,7,9} range 3, selecting the best performing value. That is, every algorithm has beentested considering each different value of k, and the best value found (the one that maximizesthe average accuracy) has been chosen.

For EF-kNN-IVFS, the set of kInit values allowed and the range in which the maximumand minimum value of m can be established have been set-up according to suggestions of [28],in which GAFuzzy-kNN was presented. The population size and the number of evaluationshave been also chosen as in [28], for the sake of a fair comparison. The rest of the parametershave been set up following the recommendations given by the authors of each technique. Thesimilarity measure used in all the experiments is the Euclidean distance.

3Note that k=1 has been excluded since most of the fuzzy nearest neighbor algorithms would becomethe 1NN rule. Also, no higher values of k have been chosen because most of the classifiers would degenerateto a majority classifier, discarding the locality capabilities of nearest neighbor algorithms

11

Table 2: Parameters configuration of the algorithms

Algorithm Reference Parameters

kNN [15] k value: 7Fuzzy-kNN [30] k value: 5, kInit: 3, m: 2D-SKNN [16] k value: 5, α: 0.95, β: 1.0IF-KNN [31] k value: 3, mA: 0.6, vA: 0.4, mR: 0.3, vR: 0.7, kInit: 3FENN [42] k value: 5 , k (edition): 5, kInit: 3IT2FKNN [12] k value: 7, kInit: {1,3,5,7,9}, m: 2GAFuzzy-kNN [28] k value: 5, kInit: 3, m: 2, Population size: 50,

Evaluations: 500, Crossover probability: 0.8,Mutation probability: 0.01

PFKNN [3] k value: 9

EF-kNN-IVFS - k value: 9, kInit range: (1,32), m range: [1,4],Sn:32, Population size: 50, Evaluations: 500

The results obtained in all the experiments will be contrasted through the use of non-parametric statistical tests [24, 23]. Specifically, we will use the Wilcoxon signed-ranks test[39] for pairwise comparisons, and the Friedman test [22] (together with the Holm procedure[27] as post-hoc) for performing multiple comparisons. More information about these statis-tical procedures specifically designed for use in the field of Machine Learning can be foundat the SCI2S thematic public website on Statistical Inference in Computational Intelligenceand Data Mining, http://sci2s.ugr.es/sicidm/.

4.2. Selection of the order operator in the voting process

The last step of the classification phase of EF-kNN-IVFS requires to provide an orderingon the intervals representing the votes assigned to each class. As in most of cases there is nodirect way of providing such order, different ordering mechanisms have been tested, aimingto improve further the classification accuracy.

The works [5, 6] elaborate on this matter, suggesting different alternatives to provide anordering between two intervals, [a, b] and [c, d]. Among those alternatives, we have selectedthe four following procedures:

• Xu: Xu and Yager order. The order is obtained by comparing a+ b and c+d. If bothsums are equal, then the expressions b− a and d− c are compared to break the tie.

• Lex-up: Lexicographic order with respect to the upper bound. The order is obtainedby comparing b and d. If both values are equal, then a and c are compared to breakthe tie.

• Lex-down: Lexicographic order with respect to the lower bound. The order is ob-tained by comparing a and c. If both values are equal, then b and d are compared tobreak the tie.

• K-order: Kα,β mapping based order. The order is obtained by comparing a+α(b−a)and c + α(d − c). If both values are equal, then a + β(b − a) and c + β(d − c) are

12

Table 3: Comparison of order operators

k = 3 k = 5

Operator Xu Lex-Up Lex-Down K-Order Xu Lex-Up Lex-Down K-Order

Average accuracy 0.8115 0.8114 0.8151 0.8123 0.8169 0.8159 0.8191 0.8165Number of wins 11 13 25 14 14 9 25 14

k = 7 k = 9

Operator Xu Lex-Up Lex-Down K-Order Xu Lex-Up Lex-Down K-Order

Average accuracy 0.8172 0.8178 0.8200 0.8183 0.8177 0.8183 0.8202 0.8173Number of wins 10 6 22 14 11 11 20 11

compared to break the tie. In our implementation, we chosen the values α = 1/3 andβ = 2/3

The full results of this study can be found in the Appendix A, and explore the accuracyobtained in test phase with each of these operators, considering four different configurationsof the k parameter (k ∈ {3, 5, 7, 9}). Table 3 shows a summary of the results obtained overthe 44 data sets, including the average accuracy and the number of wins of each algorithm(times in which it has obtained the best result).

Given the results obtained, it has been decided that EF-kNN-IVFS will incorporate thelexicographic order of the class intervals with respect to the lower bound as the operator todetermine the classification output.

4.3. Analysis of the EF-kNN-IVFS algorithm

This study has been performed to appraise the performance of EF-kNN-IVFS in contrastwith the comparison algorithms described in section 4.1. Full results of the study can befound in the Appendix A. The tables in the appendix report the average accuracy obtainedby each algorithms over the 44 datasets, the number of wins of each algorithm, and the timesover the median (times in which its performance is above the median of all the algorithmsof the experiment). A table is included per each setting of the k parameter considered(k ∈ {3, 5, 7, 9}). Also, an additional table is included showing the running time of the bestperforming setup of each algorithm in every problem.

Table 4 shows a summary of the results, considering their best performing setup. Theseresults have been contrasted by using nonparametric statistical procedures. We have chosenthe Friedman test, since it provides us with a procedure for simultaneously contrastingthe comparisons of EF-kNN-IVFS with all the techniques included in the study. As thetable shows, EF-kNN-IVFS achieves the best accuracy result of all the algorithms in thecomparison. Regarding the statistical tests, the Friedman p-value is lower than 10−10, whichmeans that significant differences are found among EF-kNN-IVFS and the rest of methods.The Holm procedure reports a p-value lower than 0.05 for the comparison between EF-kNN-IVFS and IT2FKNN, which means that the differences are significant at a α = 0.05 level.

13

Table 4: Summary results of the main comparisons

Algorithm Accuracy Friedman Rank Holm p-value Running Time

kNN 78.15 5.5000 0.00003 3.78PFKNN 78.77 5.9205 ≤ 10−5 952.21FENN 79.26 5.8523 ≤ 10−5 6.40IF-KNN 79.72 5.6477 0.00002 6.83D-SKNN 79.85 5.6477 0.00002 2.86FuzzyKNN 81.10 4.5455 0.00954 6.29GAFuzzy-kNN 81.30 4.6818 0.00674 1246.47IT2FKNN 81.11 4.3068 0.01581 13.26EF-kNN-IVFS 82.02 2.8977 - 1025.81

Friedman p-value ≤ 10−10

The p-values obtained in the rest of comparisons are lower, depicting that the differencesare significant at a α = 0.01 level, at least.

Considering these results, we can make the following analysis:

• EF-kNN-IVFS achieves a better average result than all the comparison techniqueschosen, including the evolutionary fuzzy nearest neighbor classifier (GAFuzzy-kNN).

• This good behavior can be also observed by looking at the detailed results (in theAppendix Appendix A) where EF-kNN-IVFS obtains always the highest number ofwins for each of the different settings of the k parameter. Moreover, EF-kNN-IVFS isalways the algorithm whose results are more times over the median, which shows therobustness of the method.

• The statistical analysis performed states that the differences found between EF-kNN-IVFS and the comparison algorithms are always significant, at least at a α = 0.05level. This confirms our conclusions showing that EF-kNN-IVFS is significantly moreaccurate than the rest of techniques.

• EF-kNN-IVFS running time is comparable with other evolutionary or search basedtechniques, such as GAFuzzy-kNN and PFKNN. Other comparison techniques showsa much lower running time, but it comes at a cost of a lower accuracy.

In summary, these results show us that EF-kNN-IVFS is a very competitive fuzzy nearestneighbor classifier. The use of interval values for defining memberships and computing theneighbors’ votes, together with the inclusion of the evolutionary optimization, producesbetter results than those using the classical kNN classifier, Fuzzy-kNN (fuzzy sets), D-SKNN (possibilistic classification), IF-KNN (intuitionistic fuzzy sets), IT2FKNN (type-2fuzzy sets), FENN and PFKNN (fuzzy prototype reduction techniques). The comparisonhas been completed introducing an evolutionary fuzzy nearest neighbor classifier, GAFuzzy-kNN, whose results have been also improved.

14

Table 5: Analysis of the evolutionary optimization process

EF-kNN-IVFS EF-kNN-IVFS(Non optimized) (Optimized)

Average 81.18 82.02Number of wins 13 32

Table 6: Results of the Wilcoxon test

Comparison R+ R− p-value

EF-kNN-IVFS (Optimized) vs EF-kNN-IVFS (Non optimized) 701.5 244.5 0.00509

4.4. Analysis of the evolutionary optimization process

Another study has been performed, aimed at analyzing whether the inclusion of evolu-tionary optimization for parameter optimization is effective for enhancing the performanceof EF-kNN-IVFS.

This analysis have been carried out by comparing the performance of the full EF-kNN-IVFS model with a fixed version, in which the kInit and m parameters are not determinedby the evolutionary search. Instead, a fixed configuration is setting up:

• The kInit parameter is valued in the interval (3, 9), covering the same range of valuesconsidered for the k value in the final application of the nearest neighbor rule.

• Several configuration have been tested experimentally for the m parameter. The bestone, [1.5, 2] has been chosen.

• The k value is again chosen as the best one in the {3,5,7,9} range. In this case, k = 7has been chosen.

• The rest of parameters remain the same, including the use of the lexicographic orderwith respect to the lower bound for making the final decision during the voting process.

Table 5 shows a summary of the accuracy results obtained in this study, including theaverage accuracy obtained over the 44 datasets and the number of wins of each version(full results can be found in the Appendix A). In this table, the full EF-kNN-IVFS modelis denoted as EF-kNN-IVFS (optimized), whereas the fixed, non-evolutionary version isdenoted as EF-kNN-IVFS (non optimized).

We have chosen the Wilcoxon test as a pairwise procedure for contrasting this experiment.Table 6 shows the results of the test, including the ranks obtained in each comparison (R+

and R−) and the p-value associated.The results reported in Tables 5 and 6 show that the optimized version achieves better

results, thus justifying the usefulness of the evolutionary method for determining which

15

values of the kInit and m parameters to consider during the construction of the classifier.Although a better result is not achieved in all the datasets, it can be stated that, in general,the evolutionary optimization enables to obtain a better adjusted classifier, both in thesense of average accuracy and in the number of data sets whose results have improved.These results are corroborated by the Wilcoxon test, contrasting these conclusions.

5. Conclusion

In this paper we have proposed a new evolutionary interval-valued nearest neighbor clas-sifier, EF-kNN-IVFS. IVFS are chosen as an appropriate tool for representing the instances’memberships to the different classes of the problem. They also enable our classifier to rep-resent several votes as a single interval, thus giving more flexibility to the decision rulecomputation, and ultimately, improving the generalization capabilities of the nearest neigh-bor rule. The evolutionary optimization process has eased the task of properly searching forthe best intervals of the kInit and m parameters, and thus further improving its accuracy.This has also been corroborated experimentally and it may be concluded that EF-kNN-IVFSis significantly more accurate than a selection of classical and new approaches in the stateof the art of fuzzy nearest neighbor classification.

6. Acknowledgements

Supported by the Projects TIN2011-28488, P10-TIC-06858 and P11-TIC-7765.

References

[1] J. Alcala-Fdez, A. Fernandez, J. Luengo, J. Derrac, S. Garcıa, L. Sanchez, F. Herrera, Keel data-miningsoftware tool: Data set repository, integration of algorithms and experimental analysis framework,Journal of Multiple-Valued Logic and Soft Computing 17 (2011) 255–287.

[2] J. Alcala-Fdez, L. Sanchez, S. Garcıa, M.J. del Jesus, S. Ventura, J.M. Garrell, J. Otero, C. Romero,J. Bacardit, V.M. Rivas, J.C. Fernandez, F. Herrera, KEEL: a software tool to assess evolutionaryalgorithms for data mining problems, Soft Computing 13 (2008) 307–318.

[3] M. Arif, M.U. Akram, F.A. Afsar, F.A.A. Minhas, Pruned fuzzy k-nearest neighbor classifier for beatclassification, Journal of Biomedical Science and Engineering 3 (2010).

[4] H. Bustince, Indicator of inclusion grade for interval-valued fuzzy sets. application to approximatereasoning based on interval-valued fuzzy sets, International Journal of Approximated Reasoning 23(2000) 137–209.

[5] H. Bustince, J. Fernandez, A. Kolesarova, R. Mesiar, Generation of linear orders for intervals by meansof aggregation functions, Fuzzy Sets and Systems 220 (2013) 69–77.

[6] H. Bustince, M. Galar, B. Bedregal, A. Kolesarova, R. Mesiar, A new approach to interval-valued cho-quet integrals and the problem of ordering in interval-valued fuzzy sets applications, IEEE Transactionson Fuzzy Systems 21 (2013) 1150–1162.

[7] H. Bustince, M. Pagola, E. Barrenechea, P. Melo-Pinto, P. Couto, H. Tizhoosh, J. Montero, Ignorancefunctions. an application to the calculation of the threshold in prostate ultrasound images, Fuzzy Setsand Systems 161 (2010) 20–36.

[8] D. Cabello, S. Barro, J. Salceda, R. Ruiz, J. Mira, Fuzzy k-nearest neighbor classifiers for ventriculararrhytmia detection, International Journal of Biomedical Computing 27 (1991) 77–93.

16

[9] J.R. Cano, F. Herrera, M. Lozano, Using evolutionary algorithms as instance selection for data reductionin KDD: An experimental study, IEEE Transactions on Evolutionary Computation 7 (2003) 561–575.

[10] H.L. Chen, B. Yang, G. Wang, J. Liu, X. Xu, S. Wang, D. Liu, A novel bankruptcy prediction modelbased on an adaptive fuzzy k-nearest neighbor method, Knowledge Based Systems 24 (2011) 1348–1359.

[11] M.A. Chikh, M. Saidi, N. Settouti, Diagnosis of diabetes diseases using an artificial immune recognitionsystem2 (AIRS2) with fuzzy k-nearest neighbor, Journal of Medical Systems 36 (2012) 2721–2729.

[12] F. Chung-Hoon, C. Hwang, An interval type-2 fuzzy k-nearest neighbor, in: Proceedings of the 12thIEEE International Conference on Fuzzy Systems (FUZZ-IEEE’03), St. Louis, Missouri, USA, May25-28 (2003), pp. 802–807.

[13] O. Cordon, F. Gomide, F. Herrera, F. Hoffmann, L. Magdalena, Ten years of genetic fuzzy systems:Current framework and new trends, Fuzzy Sets and Systems 141 (2004) 5–31.

[14] O. Cordon, F. Herrera, F. Hoffmann, L. Magdalena, Genetic fuzzy Systems: Evolutionary Tuning andLearning of Fuzzy Knowledge Bases, Advances in Fuzzy Systems - Applications and Theory, WorldScientific, 2001.

[15] T.M. Cover, P.E. Hart, Nearest neighbor pattern classification, IEEE Transactions on InformationTheory 13 (1967) 21–27.

[16] T. Denoeux, A k-nearest neighbor classification rule based on dempster-shafer theory, IEEE Transac-tions on Systems, Man, and Cybernetics 25 (1995) 804–813.

[17] J. Derrac, C. Cornelis, S. Garcıa, F. Herrera, Enhancing evolutionary instance selection algorithms bymeans of fuzzy rough set based feature selection, Information Sciences 186 (2012) 73–92.

[18] J. Derrac, S. Garcıa, F. Herrera, Fuzzy nearest neighbor algorithms: Taxonomy, experimental analysisand prospects, Information Sciences 260 (2014) 98–119.

[19] D. Dubois, H. Prade, An introduction to bipolar representations of information and preference, Inter-national Journal of Intelligent Systems 23 (2008) 866–877.

[20] L.J. Eshelman, The CHC adaptative search algorithm: How to have safe search when engaging innontraditional genetic recombination, in: G.J.E. Rawlins (Ed.), Foundations of Genetic Algorithms,Morgan Kaufmann, San Mateo, California., 1991, pp. 265–283.

[21] L.J. Eshelman, J.D. Schaffer, Real-coded genetic algorithms and interval-schemata, in: Proceedingsof the Second Workshop on Foundations of Genetic Algorithms. Vail, Colorado, USA, July 26-29, pp.187–202.

[22] M. Friedman, The use of ranks to avoid the assumption of normality implicit in the analysis of variance,Journal of the American Statistical Association 32 (1937) 674–701.

[23] S. Garcıa, A. Fernandez, J. Luengo, F. Herrera, Advanced nonparametric tests for multiple comparisonsin the design of experiments in computational intelligence and data mining: Experimental analysis ofpower, Information Sciences 180 (2010) 2044–2064.

[24] S. Garcıa, F. Herrera, An extension on Statistical Comparisons of Classifiers over Multiple Data Setsfor all pairwise comparisons, Journal of Machine Learning Research 9 (2008) 2677–2694.

[25] S. Greenfield, F. Chiclana, S. Coupland, R.I. John, The collapsing method of defuzzification for discre-tised interval type-2 fuzzy sets, Information Sciences 179 (2009) 2055–2069.

[26] M. Hanss, Applied Fuzzy Arithmetic. An Introduction with Engineering Applications, Springer-Verlag,2005.

[27] S. Holm, A simple sequentially rejective multiple test procedure, Scandinavian Journal of Statistics 6(1979) 65–70.

[28] X. Hu, C. Xie, Improving fuzzy k-nn by using genetic algorithm, Journal of Computational InformationSystems 1 (2005) 203–213.

[29] Y. Huang, Y. Li, Prediction of protein subcellular locations using fuzzy k-nn method, Bioinformatics20 (2004) 21–28.

[30] J.M. Keller, M.R. Gray, J.A. Givens, A fuzzy k-nearest neighbor algorithm, IEEE Transactions onSystems, Man, and Cybernetics 15 (1985) 580–585.

[31] L.I. Kuncheva, An intuitionistic fuzzy k-nearest neighbors rule, Notes on Intuitionistic Fuzzy Sets 1(1995) 56–60.

17

[32] T. Liao, D. Li, Two manufacturing applications of the fuzzy k-nn algorithm, Fuzzy Sets and Systems92 (1997) 289–303.

[33] J.M. Mendel, Uncertain Rule-Based Fuzzy Logic Systems: Introduction and New Directions, Prentice-Hall PTR, 2001.

[34] A.N. Papadopoulos, Y. Manolopoulos, Nearest Neighbor Search: A Database Perspective, Springer-Verlag Telos, 2004.

[35] J.A. Sanz, A. Fernandez, H. Bustince, F. Herrera, Improving the performance of fuzzy rule-basedclassification systems with interval-valued fuzzy sets and genetic amplitude tuning, Information Sciences180 (2010) 3674–3685.

[36] J.A. Sanz, A. Fernandez, H. Bustince, F. Herrera, A genetic tuning to improve the performance offuzzy rule-based classification systems with interval-valued fuzzy sets: Degree of ignorance and lateralposition, International Journal of Approximate Reasoning 52 (2011) 751–766.

[37] J.A. Sanz, A. Fernandez, H. Bustince, F. Herrera, IVTURS: a linguistic fuzzy rule-based classificationsystem based on a new Interval-Valued fuzzy reasoning method with TUning and Rule Selection, IEEETransactions on Fuzzy Systems 21 (2013) 399–411.

[38] G. Shakhnarovich, T. Darrell, P. Indyk (Eds.), Nearest-Neighbor Methods in Learning and Vision:Theory and Practice, The MIT Press, 2006.

[39] F. Wilcoxon, Individual comparisons by ranking methods, Biometrics Bulletin 1 (1945) 80–83.[40] D.L. Wilson, Asymptotic properties of nearest neighbor rules using edited data, IEEE Transactions on

System, Man and Cybernetics 2 (1972) 408–421.[41] X. Wu, V. Kumar (Eds.), The Top Ten Algorithms in Data Mining, Data Mining and Knowledge

Discovery, Chapman & Hall/CRC, 2009.[42] M. Yang, C. Cheng, On the edited fuzzy k-nearest neighbor rule, IEEE Transactions on Systems, Man,

and Cybernetics Part B: Cybernetics 28 (1998) 461–466.[43] L.M. Zouhal, T. Denoeux, An evidence-theoretic k-nn rule with parameter optimization, IEEE Trans-

actions on Systems, Man, and Cybernetics, Part C 28 (1998) 263–271.

Appendix A. Full results of the experimental study

The full results of the experimental study can be found below. Tables A.7, A.8, A.9and A.10 show the accuracy results obtained in the test classification phase for the differentconfigurations of the k parameter considered (k ∈ {3, 5, 7, 9}). For each data set, the bestresults obtained have been highlighted in bold. The table also includes the average accuracyobtained over the 44 datasets, the number of wins of each algorithm (times in which it hasobtained the best result) and the times over the median (times in which its performance isabove the median of all the algorithms of the experiment). Additionally, Table A.11 showsan analysis on the running time of all the algorithms (measured in seconds), consideringtheir best performing configuration.

Table A.12 shows the accuracy results obtained when analyzing the performance of EF-kNN-IVFS without the evolutionary optimization process, highlighting the best results ob-tained and including the average accuracy obtained over the 44 datasets. Note that, in thistable, the full EF-kNN-IVFS model is denoted as EF-kNN-IVFS (optimized), whereas thefixed, non-evolutionary version is denoted as EF-kNN-IVFS (non optimized).

Finally, Tables A.13, A.14, A.15 and A.16 show the accuracy results obtained whenanalyzing the four possible order operators for the voting process of EF-kNN-IVFS, followingthe same format as the previous tables.

18

Table A.7: Analysis of the EF-kNN-IVFS algorithm (k = 3)

Data sets EF-kNN-IVFS kNN Fuzzy-kNN D-SKNN IF-KNN FENN IT2FKNN GAFuzzy-kNN PFKNN

Appendicitis 84.18 84.27 87.00 84.27 87.91 87.91 87.00 84.27 82.36Balance 84.16 83.37 84.97 81.45 83.84 87.84 83.53 84.01 73.91Banana 89.26 88.64 89.06 88.62 89.40 89.28 89.09 89.19 87.74Bands 71.85 71.46 70.99 71.46 71.02 69.40 69.41 68.62 67.25Bupa 61.19 60.66 60.27 60.95 59.73 62.04 62.67 60.90 59.99Cleveland 56.64 55.14 54.94 54.80 55.58 54.92 55.61 52.22 53.29Dermatology 96.04 96.90 96.62 96.90 96.90 96.06 96.90 96.90 74.85Ecoli 83.07 80.67 82.77 80.67 82.17 82.17 82.78 82.75 79.77Glass 73.66 70.11 72.83 71.00 69.99 69.39 73.82 71.85 71.09Haberman 70.61 70.58 68.31 69.94 71.55 69.94 68.30 69.63 67.01Hayes-roth 66.25 25.00 64.38 60.00 50.00 56.25 65.00 63.13 64.38Heart 79.26 77.41 80.74 77.41 79.26 79.26 79.63 78.52 78.52Hepatitis 82.51 82.51 82.51 82.51 82.51 82.69 82.51 81.99 80.26Ionosphere 86.32 94.00 94.00 94.00 94.00 95.33 94.00 94.00 86.67Iris 94.00 85.18 84.61 85.18 84.90 84.04 84.61 84.90 79.76Led7Digit 71.60 45.20 69.60 65.20 69.80 70.40 70.00 70.60 53.80Mammographic 79.16 80.12 79.40 78.68 80.49 80.38 78.56 78.43 77.72Marketing 30.35 27.97 29.84 30.18 30.11 30.88 30.26 29.20 29.48Monk-2 84.33 96.29 78.33 96.29 96.29 79.77 79.47 81.80 72.52Movement 86.11 78.61 84.72 83.06 80.56 76.67 84.17 84.72 72.78New Thyroid 96.80 95.37 96.32 95.37 94.44 94.00 95.87 95.41 91.69Page-blocks 95.98 95.91 95.96 95.91 95.81 95.76 96.03 96.07 95.45Penbased 99.34 99.32 99.34 99.34 99.33 99.31 99.34 99.30 98.85Phoneme 90.38 88.49 89.60 88.56 88.45 88.01 89.95 89.62 86.53Pima 74.36 72.93 73.18 72.93 74.10 73.70 74.23 73.58 71.62Ring 73.77 71.82 63.07 71.82 64.38 56.24 62.41 66.32 58.81Satimage 90.88 90.94 90.61 90.96 90.68 90.24 90.77 90.41 88.27Segment 96.80 95.41 96.15 96.02 95.80 95.45 96.15 96.23 81.04Sonar 86.00 83.07 83.55 83.07 82.10 81.64 82.57 85.52 79.24Spambase 91.26 89.23 90.91 89.80 90.17 89.28 90.97 90.91 86.84Spectfheart 73.45 71.20 73.85 71.20 72.72 75.70 73.45 74.22 68.96Tae 67.67 41.13 65.62 53.75 51.08 48.38 68.29 66.29 63.04Texture 98.96 98.75 98.75 98.76 98.62 98.49 98.78 98.89 88.91Thyroid 93.99 93.89 93.92 93.88 93.92 93.75 93.96 93.89 92.78Titanic 78.83 62.61 61.66 61.66 73.65 61.66 61.66 78.69 61.61Twonorm 96.61 96.49 96.61 96.49 96.49 96.80 96.58 96.41 95.65Vehicle 70.21 71.75 70.57 71.39 70.45 69.75 70.69 69.62 64.54Vowel 99.39 97.78 98.38 97.88 97.47 96.36 98.48 99.09 83.54Wdbc 96.65 96.48 96.30 96.48 96.31 96.65 96.48 96.65 95.43Wine 95.49 95.49 95.49 95.49 95.49 95.49 95.49 95.49 93.79Winequality-red 67.42 52.41 67.54 60.35 59.66 57.78 67.85 67.23 66.79Winequality-white 67.31 48.92 66.89 59.51 57.35 56.08 67.48 66.40 66.21Wisconsin 96.82 96.38 96.82 96.38 96.53 97.11 96.67 96.67 95.34Yeast 57.75 53.17 56.74 55.73 56.88 57.21 57.82 56.54 55.80

Average 81.51 77.57 80.31 79.66 79.72 78.85 80.44 80.62 76.00Number of wins 21 4 3 5 7 10 9 4 0Times over the median 35 10 24 10 11 15 27 21 1

19





20





21





22

Table A.11: Running time (best performing versions)



Average 1025.81 3.78 6.29 2.86 6.83 6.40 13.26 1246.47 952.21

23

Table A.12: Analysis of the evolutionary optimization process

IVF-kNN EF-kNN-IVFSData sets (Non optimized) (Optimized)

Appendicitis 87.00 86.00Balance 88.95 89.28Banana 89.17 89.83Bands 69.52 69.30Bupa 63.23 65.13Cleveland 56.65 56.28Dermatology 96.34 95.78Ecoli 82.48 83.33Glass 73.50 71.34Haberman 68.96 71.25Hayes-roth 65.00 66.25Heart 80.00 79.26Hepatitis 84.67 82.66Ionosphere 84.32 85.75Iris 94.67 94.67Led7Digit 71.00 71.60Mammographic 79.88 79.63Marketing 30.92 31.03Monk-2 82.52 85.78Movement 82.50 85.00New Thyroid 95.87 96.32Page-blocks 96.13 96.22Penbased 99.20 99.38Phoneme 90.03 90.34Pima 73.06 73.71Ring 58.86 73.34Satimage 90.32 90.88Segment 96.49 96.88Sonar 82.62 85.05Spambase 91.30 91.84Spectfheart 76.05 78.30Tae 65.71 71.00Texture 98.40 99.00Thyroid 94.00 93.99Titanic 78.06 78.69Twonorm 97.03 97.19Vehicle 72.23 70.57Vowel 97.78 99.39Wdbc 97.18 96.65Wine 96.60 96.05Winequality-red 68.85 69.36Winequality-white 68.38 68.54Wisconsin 96.96 97.25Yeast 59.50 59.91

Average 81.18 82.02Number of wins 13 32

24

Table A.13: Study on the accuracy of the order operators (k = 3)

Data sets Xu Lex-Up Lex-Down K-Order

Appendicitis 0.8427 0.8327 0.8418 0.8427Balance 0.8416 0.8449 0.8416 0.8480Banana 0.8925 0.8930 0.8926 0.8930Bands 0.7123 0.7123 0.7185 0.7071Bupa 0.6086 0.6201 0.6119 0.6151Cleveland 0.5558 0.5490 0.5664 0.5593Dermatology 0.9633 0.9633 0.9604 0.9633Ecoli 0.8217 0.8336 0.8307 0.8218Glass 0.7328 0.7230 0.7366 0.7331Haberman 0.6963 0.6930 0.7061 0.6898Hayes-roth 0.6500 0.6563 0.6625 0.6500Heart 0.7926 0.7852 0.7926 0.7926Hepatitis 0.7983 0.8108 0.8251 0.7983Ionosphere 0.8460 0.8489 0.8632 0.8460Iris 0.9400 0.9400 0.9400 0.9400Led7Digit 0.7080 0.7100 0.7160 0.7100Mammographic 0.7905 0.7867 0.7916 0.7904Marketing 0.3046 0.3005 0.3035 0.3045Monk-2 0.8362 0.8406 0.8433 0.8384Movement 0.8639 0.8500 0.8611 0.8722New Thyroid 0.9677 0.9723 0.9680 0.9680Page-blocks 0.9627 0.9618 0.9598 0.9627Penbased 0.9931 0.9933 0.9934 0.9934Phoneme 0.9023 0.9004 0.9038 0.9030Pima 0.7410 0.7384 0.7436 0.7475Ring 0.7189 0.7181 0.7377 0.7186Satimage 0.9078 0.9094 0.9088 0.9091Segment 0.9654 0.9632 0.9680 0.9658Sonar 0.8457 0.8455 0.8600 0.8507Spambase 0.9134 0.9139 0.9126 0.9117Spectfheart 0.7308 0.7308 0.7345 0.7308Tae 0.6696 0.6629 0.6767 0.6696Texture 0.9898 0.9902 0.9896 0.9898Thyroid 0.9403 0.9396 0.9399 0.9401Titanic 0.7883 0.7883 0.7883 0.7883Twonorm 0.9654 0.9654 0.9661 0.9657Vehicle 0.7021 0.7033 0.7021 0.7045Vowel 0.9929 0.9909 0.9939 0.9929Wdbc 0.9665 0.9665 0.9665 0.9630Wine 0.9549 0.9549 0.9549 0.9549Winequality-red 0.6704 0.6773 0.6742 0.6785Winequality-white 0.6750 0.6727 0.6731 0.6746Wisconsin 0.9667 0.9653 0.9682 0.9667Yeast 0.5789 0.5816 0.5775 0.5755

Average 0.8115 0.8114 0.8151 0.8123Number of wins 11 13 25 14

25





26





27





28

Documents

Evolutionary Fuzzy K-Nearest Neighbors Algorithm using ...chiclana/publications/INS-D-14-1388R1.pdf · In nearest neighbor classi cation, fuzzy sets can be used to model the degree