Soft ComputDOI 10.1007/s00500-016-2094-1

FOCUS

Towards improved benchmarking of black-box optimizationalgorithms using clustering problems

Marcus Gallagher1

© Springer-Verlag Berlin Heidelberg 2016

Abstract The field of Metaheuristics has produced a largenumber of algorithms for continuous, black-box optimiza-tion. In contrast, there are few standard benchmark problemsets, limiting our ability to gain insight into the empiricalperformance of these algorithms. Clustering problems havebeen used many times in the literature to evaluate optimiza-tion algorithms. However, much of this work has occurredindependently on different problem instances and the vari-ous experimental methodologies used have produced resultswhich are frequently incomparable and provide little knowl-edge regarding the difficulty of the problems used, or anyplatform for comparing and evaluating the performance ofalgorithms. This paper discusses sum of squares cluster-ing problems from the optimization viewpoint. Propertiesof the fitness landscape are analysed and it is proposed thatthese problems are highly suitable for algorithm benchmark-ing. A set of 27 problem instances (from 4-D to 40-D),based on three well-known datasets, is specified. Baselineexperimental results are presented for the Covariance MatrixAdaptation-Evolution Strategy and several other standardalgorithms. A web-repository has also been created for thisproblem set to facilitate future use for algorithm evaluationand comparison.

Communicated by B. Xue and A. G. Chen.

M. Gallagher acknowledges the contribution of the Dagstuhl Theoryof Evolutionary Algorithms Seminar 13271 (http://www.dagstuhl.de/13271/) to the work in this paper.

B Marcus Gallaghermarcusg@itee.uq.edu.au

1 School of Information Technology and ElectricalEngineering, The University of Queensland, Brisbane 4072,Australia

Keywords Algorithm benchmarking · Continuous black-box optimization · Clustering · Fitness landscape analysis

1 Introduction

In evolutionary computation andmetaheuristic optimization,an enormous number of algorithms have been developed.Since no algorithm is superior in the theoretical, No FreeLunch sense (Wolpert and Macready 1997), in practice theperformance differences we observe depend on how wellthe mechanisms of the algorithm match the structure of theproblem landscape (Macready and Wolpert 1996). A keystep towards understanding the matching between problemsand algorithms is to develop better benchmark problemsand more rigorous approaches to the experimental analy-sis of algorithms. Unfortunately, the dominating paradigmin the literature has been to continually develop new algo-rithm variants and to evaluate these techniques in isolation.For continuous black-box optimization, artificial test func-tions (e.g. Sphere, Rastrigin, Rosenbrock) have been usedhundreds of times, but a question such as “what is the bestperformance of a black-box optimization algorithm on func-tion f , given 105 functions evaluations?” seems to be difficult(if not impossible) to answer using the literature. The situa-tion is even more problematic, because subtle differences inexperimental settings in different papers (e.g. using a differ-ent bound on the feasible search space) mean that results areoften not strictly comparable. Recently, research has begunto focus more on such experimental issues. For example,the Black-Box Optimization Benchmarking (BBOB) prob-lem set1 resolvesmany of these issues by standardisingmanyaspects of the experimental setting.However, it is very impor-

1 http://coco.gforge.inria.fr/doku.php.

123

M. Gallagher

tant to also evaluate algorithms on real-world problems, sinceit is difficult to know how well artificial test problems repre-sent real-world problems and hence to what extent algorithmperformance on artificial problems is indicative of real-worldperformance. It can be difficult to use real-world problems foralgorithm benchmarking because real problems may requireexpert domain knowledge to configure, or may come withadditional complexities that are not part of the basic optimiza-tion algorithm (e.g. complex constraints). Ideally, problemsthat are real-world “representative” while being convenientfor benchmarking should provide a valuable contribution toexperimental research practice.

This paper examines data clustering as a useful sourceof continuous, black-box benchmark problems. In Sect. 2,the sum of squares clustering problem is discussed fromthe optimization viewpoint. Some analysis of the propertiesof the fitness landscape is presented to give some intuitionfor the nature of the problem. Clustering problems havepreviously been used in the literature to test optimizationalgorithms: Section 3 examines the suitability of cluster-ing for benchmarking optimization algorithms, reviewingsome literature and discussing some limitations of previ-ously reported results. Then, in Sect. 4, a specification isproposed to describe clustering problem instances and a setof problem instances defined (and made available via theweb). To establish some baseline results for future compari-son, a number of commonly used algorithms are applied tothe clustering problem sets. The experiments are describedin Sect. 5 and results presented in Sect. 6. Where possible,the results are also compared with previous results from theliterature, revealing some surprising insights. The work issummarised in Sect. 7. This paper is an extended version ofa paper presented at the SEAL 2014 Conference (Gallagher2014), with significant extensions to the landscape analysisand discussion of clustering problems and with additionalresults for the CMA-ES algorithm.

2 Clustering

The (continuous) sum of squares clustering problem (see,e.g. Xu II and Wunsch 2005) can be stated as follows: Givena set X = {x1, . . . , xn} ⊆ IRd of n data points, determinethe locations of a set of k cluster centres in the data space,C = {c1, . . . , ck} ∈ IRd to minimize

f (C|X ) =n∑

i=1

k∑

j=1

bi, j‖xi − c j‖2, (1)

where

bi, j ={1 if ‖xi − c j‖ = min j ‖xi − c j‖0 otherwise.

(2)

and ‖ · ‖ is the Euclidean/L2 metric. The problem variablesare the coordinates of the cluster centres in the data space.Denote the d-dimensional coordinates of the i th cluster cen-tre as ci = (yi , yi+1, . . . , yi+d); then the problem can berewritten as an unconstrained, continuous optimization prob-lem of dimensionality dk:

min f (y), y ∈ IRdk (3)

A specific clustering problem instance is, therefore, definedby a dataset, X and a value of k. The dimensionality of theunderlying optimization problem grows in proportion to dand k, but is independent of the number of data points, n.Figure 1 shows an example of a clustering problem (andsolution), where n = 75, d = 2 and k = 4.

An equivalent problem to clustering from operationsresearch is a specific case of the (continuous, uncapacitated)location-allocation problem, also known as facility locationor multisource Weber problem (Brimberg et al. 2000; Salhiand Gamal 2003). Given a set of customers to be servicedby a set of facilities, the problem is to position the facilitiesto optimize a criterion measuring overall service, under thefollowing conditions:

– the set of customer locations is given by X ,– the set of facility locations be C,

0 20 40 60 80 100 1200

20

40

60

80

100

120

140

160

x1

x 2

Fig. 1 The Ruspini dataset (datapoints shown as “+”), with two possi-ble solutions for the clustering problem with k = 4. The cluster centresshown as blue circles represent the global optimum ( f () ≈ 12881.051).A second possible solution (found by a run of the k-means algorithm)is shown as red squares ( f () ≈ 49778.908) (color figure online)

123

Towards improved benchmarking of black-box optimization algorithms using clustering…

– assuming equal customer weightings, unlimited capacityof facilities to provide service and Euclidean distancesbetween customers and facilities;

the problem then reduces to the sum of squares clusteringproblem. Location-allocation problems occur in many real-world applications involving the planning and placement ofentities such as public facilities, warehouses and retail stores(Brimberg et al. 2000).

Clustering is a fundamental task in machine learning,statistics and data analysis that has been the subject of alarge amount of study (see, e.g. Jain et al. 1999; Xu II andWunsch 2005 for reviews). The Euclidean/L2 metric is themost commonly used objective function, though variationsof the problem are formulated by considering alternative dis-tance metrics (e.g. using the L1 norm results in k-mediansclustering). Finding a global optimum to the underlyingoptimization problem is known to be NP-hard, even in therestricted cases where d = 2 or k = 2 (Vattani 2011). Sincethe ordering of cluster centres in a solution vector is arbi-trary, it is also known that the error/fitness landscape of theoptimization problem is comprised of k! equivalent “wedge”corresponding to permutations of the labels of the cluster cen-tres, c j . A analogous effect is seen inmixturemodels [knownas label switching (Stephens 2000)] and in multi-layer per-ceptron neural networks (Gallagher 2000; Hecht-Nielsen1990).

To gain some further insight into the landscape of clus-tering problems, consider first a simple example where p =1, n = 2 and k = 2 with y1 = −1, y2 = 1. The dimen-sionality of the problem is 2, so the landscape can be vieweddirectly. Figure 2 (Left) shows the symmetry of the spaceacross the line c1 = c2, with a quadratic bowl shape oneither side due to the distance metric. When a cluster centre

is positioned at each data point, f (c1, c2) = 0. In the range[−1, 1], the worst solution is when both cluster centres areat one of the data points, leading to f (c1, c2) = 4. Figure 2(Right) shows that even for this very simple problem, thelandscape has more structure when the range is extended,e.g. to [−3, 3]. Additional folds are present in the landscapeat values where the assignment of the closest cluster centre toa datapoint changes. A more complicated example probleminstance (used in this paper) is given using the Iris dataset(p = 4, n = 150) with k = 7, i.e. a 27-D optimizationproblem.

The k-means algorithmwas run on this problem, converg-ing to a candidate solution after some number of iterations.From this point on the landscape,walkswere conducted (withsmall stepsize) parallel to each individual coordinate axis andholding all other solution variables constant. Figure 3 shows aparallel coordinates plot of the result, with the candidate solu-tion point shown with circles. The landscape appears to bea combination of many intersecting quadratic bowls, ridgesand plateaus. The structure of a particular problem instancewill depend on the distribution of the dataset. Figure 4 zoomsin on a region of the same plot, revealing more structure atsmaller scales. Finally, Fig. 5 zooms in even further, revealingthat the solution found by k-means in this particular trial isnot a global optimum, since it can be seen that perturbing thesolution along one variable discovers a basin of better solu-tions. Note that these plots only show 1-D slices through thehigh-dimensional landscape, so other structure on the land-scape might not be evident here. In particular, points thatappear to be local optima along these slices may in fact besaddle points.

A large number of algorithms have been proposed for clus-tering, though there is little doubt that the k-means algorithmis the most widely known and used (Steinley 2006). From

Fig. 2 Fitness landscapes for a simple clustering problem with two data points, y1 = −1, y2 = 1. Left The landscape in the range [−1, 1]. RightThe same landscape with the range extended to [−3, 3]

123

M. Gallagher

Fig. 3 Parallel coordinates plotin a 27-D clustering problemlandscape

Fig. 4 A zoom-in on a regionof the landscape from Fig. 3

an optimization perspective, k-means is a local iterative opti-mization algorithmwhich follows a non-increasing trajectoryover f . It is not a black-box algorithm; nevertheless its pop-ularity makes it frequently used in experimental comparativestudies. Note also that solving the optimization problem (i.e.locating cluster centres) is often not the final goal of clus-tering. Further analysis might include studying which datapoints are assigned to which cluster centre, or producing aclassifier, where each cluster represents a class in the data set

and the class label of future data points can then be predicted(e.g. using the minimum distance from the cluster centres).

3 Clustering problems for black-box optimizationbenchmarking

Clustering problems have a number of properties which sug-gest that they might provide an extremely useful source of

123

Towards improved benchmarking of black-box optimization algorithms using clustering…

Fig. 5 A closer zoom-in on aregion of the landscape fromFigs. 3 and 4

benchmark problems for the evaluation and comparison ofalgorithms:

– They seem to be generally challenging to solve.– They are scalable in dimensionality (via d and k).– They are “real-world” problems in data analysis (i.e.datasets can come from real-world problems).

– They are unconstrained, meaning that black-box algo-rithms can be readily applied without the need for aconstraint-handling mechanism.

– They can be implemented relatively simply and donot require a large amount of problem-domain-specificknowledge to understand.

– The objective function is not expensive to evaluate.

There are currently few (if any) benchmark problem sets forcontinuous, black-box optimization that have all of the aboveproperties. This suggests an exciting opportunity to improveon and increase the utility of experimental black-box algo-rithm evaluation and comparison by building a standardisedset of clustering problem instances.

Given the fundamental nature of the clustering problemand data analysis, it is not surprising that hundreds of clus-tering algorithms have been proposed in the literature. At thesame time, general-purpose metaheuristics and other opti-mization techniques have also been applied to clusteringproblems. This paper does not attempt an exhaustive reviewof all this work, but rather aims to extract the important issuesto be considered in developing a specification of clusteringproblems for black-box optimization benchmarking.

3.1 Difficulties in comparing with previous results

One of the major difficulties in trying to compare an algo-rithmwith previouswork stems from the absenceof standardsin the way results have been presented. Clustering results arepresented in a variety of ways in the literature (Xu II andWunsch 2005). While the sum of squares objective functionis frequently used, the actual function values obtained andthe number of evaluations made by the algorithm are some-times not reported. Instead, measures of cluster shape aroundthe cluster centres produced have been used [e.g. the Randindex is used by Chang et al. to evaluate their genetic algo-rithm variant (Chang et al. 2009)]. Kanade and Hall presenta fuzzy ant clustering method, but use a reformulated (fuzzi-fied) objective function and only present their results in termsof the reformulated function (Kanade and Hall 2007). Whenthe intended application is classification, measures such asclassification accuracy on the data are used [e.g. Liu et al.(2010) evaluate a fuzzy C-means, genetic algorithm-basedfuzzy C-means and an immunodominance clonal selectionfuzzy C-means algorithm in this way].

While clustering problems have been widely used tocompare algorithms, the datasets that have been used havealso varied widely from paper to paper. Some authors gen-erate artificial datasets with known structure/distributions;however, these datasets are not usually made available. Ifsufficient details are given, the data could be regenerated;however, this information is not always available. There arealso a large number of benchmark datasets available in themachine learning research community, and different authors

123

M. Gallagher

select different datasets to use. Focussing on specific datasetswould clearly improve the comparability of results for black-box optimization algorithms.

Finally, there are many experimental factors that are notspecific to clustering problems that can impact on future com-parisons of results. The fundamental performance results arein terms of the best objective function values found (or sta-tistics of such values over multiple trials) and the number offunction evaluations used. Presenting results in figures hasseveral advantages, but on the other hand it is often difficultto read off numerical values from a graph for comparison(cf. a result from Likas et al. 2003 discussed in Sect. 6.1).Full details of the experimental configuration (e.g. algorithmparameter settings, termination criteria, number of repeatedtrials) are essential to permit fair comparison and reproduc-tion of results (Hooker 1996; McGeoch 2002; Rardin andUzsoy 2001).

3.2 Results selected for comparison

The literature was further reviewed for experimental resultsthat could be compared in a black-box optimization context.A representative number of approaches were identified:

– Maulik and Bandyopadhyay (2000) develops a real-valued genetic algorithm (GA) for clustering and com-pares with k-means.

– Ye and Chen (2005) apply particle swarm optimization(PSO), ant colony optimization (ACO) and a honey beesalgorithm.

– Kao and Cheng (2006) develop an ant colony optimiza-tion algorithm and compare it with a previous ACOclustering algorithm (due to Shelokar et al. 2004) andk-means.

– Fathian et al. (2007) present a honey bee mating algo-rithm and compare it with ACO, a GA, Tabu search (TS)and simulated annealing (SA).

– The hyperbolic smoothing (HSC) method presented inXavier (2010).

– Taherdangkoo et al. (2013) propose a blind, naked mole-rats algorithm and compare it with k-means, two GAvariants, PSO, ACO, simulated annealing and artificialbee colony algorithms.

– Hatamlou (2013) propose a black-hole inspired algorithmand compare it with k-means, particle swarm optimiza-tion (PSO), a gravitational search algorithm (GSA) andthe big bang crunch algorithm (BB-BC).

– Xiang et al. (2015) present a dynamic differential evo-lution algorithm (DSDE) and compare it with twoother differential evolution algorithms (DE/rand/1/binand DE/best/1/bin).

– Berthier (2015) presents a progressive differential evo-lution (PDE) algorithm and compares results with adifferential evolution (DE) implementation.

These papers have each used different datasets to evaluateand compare algorithms. One problem instance is commonacross all the papers—these results are compared in Sect. 6.1.

4 Clustering problem instances

As mentioned above, many different datasets have been uti-lized to evaluate and compare clustering algorithms, suchas those from the UCI Machine Learning Repository (Blakeet al. 1998). Of particular relevance, Du Merle et al. (2000)used an interior point algorithm to compute approximateglobal optimum values for problem based on the Iris, Ruspiniand German Towns (Spath) datasets. This is useful becausewe can assess the performance of algorithms relative to theoptimal value on these problems. These datasets have alsobeen used in other papers. It is proposed to use them as aninitial benchmark set.

4.1 Clustering problem specification

To be useful for black-box optimization evaluation, a cluster-ing problem should be specifiedwith the following elements:

1. A dataset, X of dimensionality d.2. A value of k.3. An initial bounded search space, which contains the

global optimum. This can be done using the minimumand maximum value in the dataset as the upper and lowerbounds of the search space. For simplicity the overallminimum and maximum are used for every variable. Atighter search space could consider the minimum andmaximum of each variable independently; however. theimplementation would be more complex.

Subsequently, the set of 27 problem instances proposed hereis specified as follows:

– The Iris dataset, with d = 4, k = 2, . . . , 10 and initialsearch space [0.1, 7.9]dk .

– The Ruspini dataset, with d = 2, k = 2, . . . , 10 andinitial search space [4, 156]dk .

– The German Towns dataset, with d = 3, k = 2, . . . , 10and initial search space [24.49, 1306024]dk .

To facilitate future use of these problems, a web reposi-tory has been created at http://realopt.uqcloud.net/crwr.html.The repository records the specifications of each problem

123

Towards improved benchmarking of black-box optimization algorithms using clustering…

instance, the global optimum (solution vector and objec-tive function value) and a copy of the dataset. This will beextended to record results on these problems from the liter-ature.

5 Experimental details

To make a comparison and establish some results for theselected clustering problems, the following algorithms wereevaluated:

– CMA-ES: the Matlab implementation of the CovarianceMatrix Adaptation Evolution Strategy available fromhttps://www.lri.fr/~hansen/cmaes_inmatlab.html wasused with default parameter settings (as recommended,the initial search variance was set to 2/3 of the searchspace.

– CMA-ES (50,100): the same implementation of CMA-ES but with a (larger) population size of 50.

– NM: the Nelder–Mead simplex algorithm, as imple-mented in the Matlab fminsearch function. Defaultparameter settings were used, with termination crite-ria extended so that the algorithm ran until a toleranceof change in variables or function values was less than10−10, or if 3 × 105 function evaluations were reached.The algorithm is initialized at a randompoint in the searchspace for each trial.

– RS: uniform random search over the search space. Thealgorithm was given 105 function evaluations.

– KM: the k-means clustering algorithm. Cluster centreswere initialized to be randomly selected data points (thisis probably themost commonmethod in the literature butthere are many other possibilities Steinley 2006).

The algorithms were chosen first because they can be appliedwith little setting of internal parameters. In addition, CMA-ES is a well-regarded evolutionary black-box algorithm, NMis the standard Matlab solver, KM is a standard non-black-box clustering algorithm and RS provides a useful baseline.For all algorithms, solution vectors were represented assuggested by the problem formulation in Sect. 2, i.e. thecoordinates of each cluster centre appear with coordinatevalues in the same order and concatenated together into adk-dimensional continuous vector. Note, however, that apartfrom KM, all the algorithms are black-box and operate with-out the knowledge of the clustering problem that is implicitin the representation. This means, for example, that if a ran-dom perturbation was applied to the order of vector elements(which could be “unencrypted” before each function evalua-tion), the performance of the black-box algorithms would beunaffected.

Each algorithm was run for 50 restarts. Note that thedifferent algorithms ran for different numbers of functionevaluations. The intentionwas not to impose a fixed budget offunction evaluations across the algorithms but to allow themto use the amount of resources they request to“converge”.Results on these problems can be of interest for any rea-sonable budget of function evaluations. Future research maychoose to focus on a “limited budget” scenario or on find-ing high-quality solutions using a possibly large number offunction evaluations. Different algorithm specifications willbe more suitable to different budgets of function evaluationsand any result that improves upon previous results makes aworthwhile contribution.

6 Results

The experimental results are shown in Tables 1, 2 and 3.Overall, CMA(50,100) gave the best performance, with aver-age values that were closer to the optimal value that the othertechniques. It required between 10,000 and 50,000 functionevaluations. CMA used a smaller population size (deter-mined automatically) and used between 2000 and 25,000function evaluations. The results were very similar on someproblems (e.g. for Ruspini n = 4, 6, 10) but an order ofmagnitude worse on others (e.g. German, n = 8−20). TheNM results are considerably worse across the problem setsthan CMA(50,100) and worse than CMA for the Iris andRuspini problems, but (interestingly) better on the GermanTowns problems. As a local search algorithm, it does, how-ever, use a much smaller budget of function evaluations:between 1000 and 8000. As a completely non-local algo-rithm, RS outperforms the standard Matlab solver (NM)on most of the Ruspini problem instances! Finally, KMas a non-black-box solver has a considerable advantageover the other algorithms. It converges very quickly, tak-ing less than 20 iterations/function evaluations across theproblems tested. Its performance is relatively good; however,CMA(50,100) still provides better performance on all prob-lem instances! This is an impressive result for a black-boxsolver and experimentally demonstrates that metaheuristicsare able to outperform problem-specific algorithms and thatglobal/population-based search would seem to lead to resultsthat are difficult to obtain with a local/trajectory-based algo-rithm.With such small requirements for function evaluations,large amounts of restarts could be performed for KM. Never-theless, the results here over 50 runs at least indicate that thefitness landscapes of clustering problems contain structurethat causes problems for the standard solver in this problemdomain.

Tables 1, 2 and 3 also show results for statistical sig-nificance tests done between the results for all pairs ofalgorithms on each problem. Table 4 gives a legend for the

123

M. Gallagher

Table 1 Results for clusteringproblems

D n f ∗ CMA(50,100) CMA(50,100) # f

I 8 1.52348e02 1.523480e02(0.0e00)β 1.1168e04(2.9e02)

12 7.88514e01 7.885144e01(0.0e00)β 1.7136e04(5.7e02)

16 5.72285e01 5.922328e01(5.0e00)β 2.1938e04(7.4e02)

20 4.64462e01 4.904783e01(2.3e00)β 2.5092e04(9.5e02)

24 3.90400e01 4.076697e01(3.5e00)β 2.7466e04(1.3e03)

28 3.42982e01 3.554887e01(1.4e00)β 3.0220e04(1.1e03)

32 2.99889e01 3.232285e01(2.1e00)β 3.3250e04(9.3e02)

36 2.77861e01 2.946070e01(1.9e00)β 3.6432e04(1.4e03)

40 2.58341e01 2.705470e01(1.3e00)β 3.9214e04(1.5e03)

R 4 8.93378e04 8.93378e04(0.0e00)αγ δ 1.3842e04(4.2e03)

6 5.10635e04 5.11278e04(4.4e01)αβδ 1.9382e04(2.9e03)

8 1.28811e04 1.28811e04(0.0e00)αδ 1.6822e04(5.1e03)

10 1.01267e04 1.10334e04(6.3e02)αδ 2.0862e04(4.9e03)

12 8.57541e03 8.84859e03(6.8e02)α 2.2022e04(1.1e03)

14 7.12620e03 7.55527e03(7.4e02)α 2.5912e04(2.3e03)

16 6.14964e03 6.43566e03(3.5e02)αβ 2.8792e04(2.7e03)

18 5.18165e03 5.64378e03(2.0e02)α 3.1892e04(4.3e03)

20 4.446.28e03 4.80930e03(2.5e02) 3.1822e04(2.4e03)

G 6 6.02546e11 6.02547e11(0.0e00)β 1.6282e04(4.7e03)

9 2.94506e11 3.08336e11(2.9e10)βγ 2.2952e04(5.5e03)

12 1.04474e11 1.42481e11(8.0e10)β 2.8652e04(8.1e03)

15 5.97615e10 7.46629e10(1.5e10)β 3.2272e04(5.0e03)

18 3.59085e10 4.80932e10(9.0e09) 3.3402e04(1.5e03)

21 2.19832e10 4.40172e10(9.4e09) 3.9092e04(2.6e03)

24 1.33854e10 3.11688e10(1.2e10) 4.0962e04(2.6e03)

27 7.80442e09 2.26611e10(1.2e10) 4.6382e04(3.5e03)

30 6.44647e09 2.80614e10(1.2e10) 4.9872e04(5.8e03)

Problem instances are defined by a dataset (D), i.e. Iris (I), Ruspini (R) or German towns (G), together withthe problem dimensionality (n), where n = dk. Shown are mean and standard deviations over 50 trials ofeach algorithm. The mean and average number of function evaluations (# f ) is also shown for eachalgorithm. A result in bold font means that if one or more of the 50 trials located the global optimum ( f ∗) toat least 15 significant figures

symbols used. For example, on the Iris dataset, the meanperformance of CMA(50,100) is not statistically signifi-cant from that of KM (indicated by the superscript β forthese rows of results in Table 1 and the matching rowsfor KM in Table 3. On the Ruspini problem with n = 4,multiple symbols show no significant different between themeans ofCMA(50,100) v’sCMA,CMA(50,100) v’sKMandCMA(50,100) v’s RS. But for Ruspini with n = 20, themeanperformance of CMA(50,100) was significantly differentfrom all other algorithms, since no symbols are shown. Over-all, the majority of differences reported show significantlydifferent mean performances. It should also be noted that formany pairs of results where themean difference is not signifi-cant, the standard deviations of the algorithm’s performancescan be very different, which is also an important practicalconsideration.

On most problems (shown with bold), CMA(50,100)found the global optimum on at least one of the 50 trials. Theexceptions were some of the larger problems on the GermanTowns problems. It is an open question to establish resultson these problems to see howmany functions evaluations arerequired to locate the global optimum. KM finds the globaloptimum for around half of the problems (lower dimensions)and CMA and NM do so for some of the smaller probleminstances.

Figures 6, 7 and 8 compare the average fitness perfor-mance of the algorithms over the problems from each dataset.Results are given as a performance ratio with the globaloptimum value for each problem instance. The trends asproblem dimensionality increases give some indication ofthe scaling behaviour of the performance of each algorithm.As expected, random search steadily increases (given a fixed

123

Towards improved benchmarking of black-box optimization algorithms using clustering…

Table 2 Results for clusteringproblems

D n CMA CMA # f RS

I 8 2.68733e02(2.2e02)η 2.41780e03(2.1e02) 2.93656e02(4.1e01)η

12 1.96530e02(1.8e02)η 3.77214e03(5.3e02) 2.60478e02(3.0e01)η

16 1.26181e02(8.8e01) 4.98944e03(5.9e02) 2.39601e02(2.8e01)ν

20 1.20999e02(8.9e01) 6.58952e03(9.8e02) 2.2239e02(2.7e01)

24 1.06623e02(8.9e01)ε 8.39298e03(1.3e03) 2.1431e02(1.9e01)ν

28 8.90713e01(3.3e01)ε 9.90306e03(1.2e03) 2.0180e02(2.4e01)

32 8.84456e01(3.3e01) 1.20515e04(1.8e03) 1.9722e02(1.8e01)ν

36 8.38895e01(2.9e01) 1.46102e04(2.4e03) 1.9432e02(1.9e01)ν

40 8.00406e01(3.2e01) 1.67537e04(3.0e03) 1.8527e02(1.8e01)

R 4 8.93378e04(0.0e00)αζη 1.4188e03(3.0e02) 9.01376e04(4.0e02)δην

6 5.11094e04(4.8e01)αεη 3.2726e03(3.2e02) 5.37281e04(1.0e03)δηλ

8 1.66519e04(1.2e04)αη 2.8380e03(7.9e02) 1.99433e04(2.3e03)δη

10 1.14295e04(1.2e03)αη 3.1720e03(2.2e02) 1.65256e04(1.6e03)δηλ

12 9.86935e03(9.6e02)αη 3.8949e03(2.1e02) 1.50499e04(1.0e03)ηλ

14 8.76290e03(1.4e03)αεη 4.9135e03(7.2e02) 1.36176e04(1.0e03)ηλ

16 7.75958e03(1.5e03)αεη 5.7632e03(6.7e02) 1.24834e04(1.0e03)ηλ

18 6.67363e03(1.1e03)αεη 6.1772e03(7.1e02) 1.15749e04(7.4e02)ην

20 6.87915e03(1.4e03)α 7.2632e03(1.5e03) 1.09593e04(7.1e02)

G 6 1.55257e12(8.2e11)ζ 3.6002e03(4.5e02) 1.08704e12(1.6e11)

9 1.04493e12(8.0e11)η 5.4150e03(8.9e02) 9.12653e11(1.3e11)η

12 1.05834e12(7.8e11) 7.3940e03(1.7e03) 8.50900e11(9.5e10)η

15 8.33722e11(7.2e11)η 1.0106e04(2.4e03) 7.83531e11(8.3e10)

18 5.99661e11(6.0e11)η 1.19768e04(1.4e03) 7.50905e11(9.0e10)η

21 4.31552e11(1.5e11) 1.44593e04(1.8e03) 6.66544e11(8.9e10)

24 4.17723e11(1.6e11)ζ 1.82163e04(1.4e03) 6.67006e11(8.7e10)

27 4.05634e11(1.8e11) 2.09944e04(1.9e03) 6.50674e11(8.4e10)

30 4.19464e11(1.7e11) 2.3522e04(3.2e03) 6.34447e11(8.7e10)

Problem instances are defined by a dataset (D), i.e. Iris (I), Ruspini (R) or German towns (G), together withthe problem dimensionality (n), where n = dk. Shown are mean and standard deviations over 50 trials ofeach algorithm. The mean and average number of function evaluations (# f ) is also shown for eachalgorithm. A result in bold font means that if one or more of the 50 trials located the global optimum ( f ∗;Table 1) to at least 15 significant figures

budget of function evaluations). Some individual probleminstances also appear to be particularly challenging for thealgorithms tested. For example, the 8D Ruspini problemappears more difficult for KM and CMA, than both the 6Dand the 10D Ruspini problem instances (Fig. 7). Note alsothat the German towns problems lead to poor performanceratios for the algorithms compared to the other datasets (inFig. 8 the y axis is on a log scale).

The comparative results in Figs. 6, 7 and 8 give a gen-eral indication of performance, but it is important to notethat average fitness values are a relatively gross summaryof the results and may hide important details. For example,the average performance of CMA on the 12D and 15D Ger-man towns problems (Fig. 8) is slightly worse than randomsearch! However, Table 1 shows that the standard deviationof CMA results on these problems is relatively large. Further

investigation of these results revealed that many trials foundsolutions much better than the average, but a number of othertrials converged to a poor solution considerably worse. Theaverage is a poor summary of such results.

Some further useful observations can be made by lookingat the performance curves/progress made by an algorithmduring execution.TheCMA(50,100) algorithmwas run againon two of problem instances: the 10D Ruspini and 30D Ger-man towns (50 new trials per instance). The results (in termsof the best-so-far solution found at a given time) are shown inFigs. 9 and 10. The algorithm converges towards the globaloptimum on over half of the trials for the 10D Ruspini prob-lem (Fig. 9), while the remaining trials seem to converge toa small number of distinct, worse solutions. Again, it is clearthat summarizing these values with themean hides importantinformation. For the 30D German towns problem (Fig. 10),

123

M. Gallagher

Table 3 Results for clustering problems

D n NM NM # f KM KM # f

I 8 5.1208e02(2.5e02) 1.0938e03(5.8e02) 1.52348e02(0.0e00)β 4.66e00(1.4e00)

12 3.7454e02(2.6e02) 1.6421e03(6.1e02) 8.79682e01(2.3e01)β 6.94e00(2.9e00)

16 3.0047e02(2.4e02)ρ 2.0770e03(5.5e02) 6.24359e01(6.9e00)β 8.2e00(3.7e00)

20 2.6579e02(2.2e02) 2.6568e03(8.8e02) 5.37562e01(8.8e00)β 9.18e00(3.6e00)

24 2.6285e02(2.3e02)ρ 3.0913e03(1.3e03) 4.66559e01(8.7e00)βε 7.86e00(2.9e00)

28 2.6404e02(2.2e02) 3.8847e03(2.8e03) 4.09041e01(5.5e00)βε 8.82e00(3.4e00)

32 1.7233e02(1.3e02)ρ 4.3597e03(1.4e03) 3.70879e01(7.5e00)β 8.34e00(2.8e00)

36 1.6729e02(1.1e02)ρ 5.0430e03(2.8e03) 3.39236e01(3.8e00)β 7.78e00(2.1e00)

40 1.3556e02(3.2e01) 5.8026e03(2.3e03) 3.09606e01(1.0e01)β 8.52e00(3.0e00)

R 4 8.9338e04(0.0e00)γ ζρ 5.03e02(2.5e01) 9.67212e04(2.3e04) 3.82e00(1.5e00)

6 6.6428e04(2.1e04) 8.004e02(1.3e02) 5.11096e04(4.6e01)βελ 3.70e00(1.5e00)

8 3.5846e04(2.1e04) 1.275e03(3.3e02) 2.83654e04(1.8e04) 3.99e00(1.6e00)

10 3.5366e04(2.1e04) 1.9712e03(1.3e03) 1.86163e04(1.5e04)λ 4.45e00(1.6e00)

12 2.7332e04(2.0e04) 4.203e03(2.8e03) 1.46650e04(1.2e04)λ 4.66e00(1.7e00)

14 2.6147e04(2.0e04) 7.4512e03(2.8e03) 1.07800e04(7.7e03)ελ 5.08e00(1.7e00)

16 2.5967e04(2.1e04) 7.5234e03(4.4e03) 9.58363e03(6.8e03)βελ 5.00e00(1.7e00)

18 2.4995e04(2.1e04)ρ 1.1427e04(3.9e03) 8.22721e03(5.1e03)ε 5.01e00(1.7e00)

20 1.0728e04(1.2e03) 5.9614e03(2.1e03) 6.97757e03(3.9e03) 5.30e00(1.7e00)

G 6 1.5526e12(8.7e11)ζ 9.908e02(4.1e02) 6.51273e11(1.6e10)β 5.95e00(1.4e00)

9 4.1772e11(1.7e11)γ θ 2.8568e03(1.4e03) 3.63195e11(3.7e09)βθ 9.07e00(1.5e00)

12 5.7125e11(1.7e11) 2.1122e03(6.1e02) 2.73230e11(3.1e10)β 1.51e01(2.8e00)

15 4.9316e11(1.5e11) 3.0132e03(1.8e03) 2.49510e11(4.4e10)β 1.55e01(3.2e00)

18 3.3045e11(1.8e11)θ 7.8998e03(7.5e03) 2.29057e11(6.0e10)θ 1.53e01(5.3e00)

21 3.2013e11(3.5e10) 5.0868e03(4.4e02) 2.09148e11(8.1e10) 1.43e01(4.2e00)

24 4.5023e11(1.8e11)ζ 4.2644e03(1.4e03) 1.79267e11(9.8e10) 1.39e01(4.3e00)

27 2.8713e11(2.1e11) 7.598e03(4.1e03) 1.35754e11(1.1e11) 1.33e01(3.6e00)

30 2.7033e11(9.7e10) 6.4218e03(1.5e03) 1.10532e11(1.0e11) 1.32e01(4.0e00)

Problem instances are defined by a dataset (D), i.e. Iris (I), Ruspini (R) or German towns (G), together with the problem dimensionality (n), wheren = dk. Shown are mean and standard deviations over 50 trials of each algorithm. The mean and average number of function evaluations (# f ) arealso shown for each algorithm. A result in bold font means that if one or more of the 50 trials located the global optimum ( f ∗; Table 1) to at least15 significant figures

Table 4 Legend for the significance tests between the results presentedin Tables 1, 2 and 3

CMA KM NM RS

CMA(50,100) α β γ δ

CMA ε ζ η

KM θ λ

NM ν

The symbols shown indicate no significant difference (using a 95%con-fidence interval) between the mean performance result of each pair ofalgorithms. All pairs of algorithms were tested using one-way ANOVAwith a Tukey Post-hoc test

a much larger number of suboptimal convergence values areobserved. Interestingly, a single runout of 50 converged to theglobal optimum. In the comparative results reported above,

no runs found of this algorithm found the global optimum(giving us a 1 % empirical success rate!). This demonstratesthat conclusions drawn from experiments need to always becarefully moderated with the evidence they produce. For thisexample, infrequent events may be missed if a small numberof repeated trials is performed.

6.1 Comparison with previous results

Table 5 shows the results reported by previous papers forone of the problem instances tested, Iris with k = 3. A largevariety and number of algorithms have been tested on thisproblem. Comparing these results with those from above,the most striking thing is that the majority of these reportedresults are relatively poor. The average objective function val-ues are far from optimal and are significantly outperformed

123

Towards improved benchmarking of black-box optimization algorithms using clustering…

Fig. 6 Performance results(mean best fitness) for the Iris(I) dataset problems as a ratiowith the globally optimal value(e.g. a value of 2.5 means theaverage best solution found byan algorithm was 2.5 times thevalue of the global optimum

8 12 16 20 24 28 32 36 400

2

4

6

8

Problem Dimensionality (n)P

erfo

rman

ce R

atio

CMA(50,100)CMAKMNMRS

Fig. 7 Performance results(mean best fitness) for theRuspini (R) dataset problems, asa ratio with the globally optimalvalue

4 6 8 10 12 14 16 18 200

1

2

3

4

5

Problem Dimensionality (n)

Per

form

ance

Rat

io

CMA(50,100)CMAKMNMRS

Fig. 8 Performance results(mean best fitness) for theGerman towns (G) datasetproblems, as a ratio with theglobally optimal value

6 9 12 15 18 21 24 27 3010

0

101

102

Problem Dimensionality (n)

Per

form

ance

Rat

io

CMA(50,100)CMAKMNMRS

by CMA(50,100). These results tend to be based on fewerfunction evaluations, but the papers do not seem to be target-ting a “low budget” scenario, rather evaluating the potentialof the algorithms. Another significant anomaly is the differ-ences between the k-means (KM, SBKM) results reportedin these papers (average values between 97 and 101) andthe result obtained in this paper for KM on this problem(8.79682e01(2.3e01)). There may be a difference in the ini-tialization technique used which is not mentioned in any ofthese papers. Likas et al. (2003) also use the Iris (k = 3) prob-lem to evaluate their global k-means algorithm; however, the

results are reported on a graph so the exact numbers are notavailable. Their graph indicates the original k-means algo-rithm has an average objective function value around 105.It is not clear why this result is so poor: initialization mayagain be a cause. The global k-means algorithm appears toachieve an average value close to the global optimum fromthe graph.

The hyperbolic smoothing (HSC) method presented inXavier (2010), achieves a result close to the global optimum(Table 5). This is not a black-box method and the numberof function evaluations used is not reported (the algorithm

123

M. Gallagher

Fig. 9 50 repeated trials of theCMA(50,100) algorithm on the10D Ruspini problem instance.Curves are best-so-far objectivefunction values. The value of theglobal optimum is also shown

Fig. 10 50 repeated trials ofthe CMA(50,100) algorithm onthe 30D German towns probleminstance. Curves are best-so-farobjective function values. Thevalue of the global optimum isalso shown

converges smoothly towards the optimum and the algorithmterminated when a threshold of accuracy was reached). Thevery recent paper by Berthier (2015) actually takes the entireset of benchmark functions proposed here (taken from Gal-lagher 2014) and achieves very good results overall and inparticular on this Iris (k = 3) problem instance. Variantsof differential evolution were used, but the results are very

different from the (also very recent) results in Xiang et al.(2015) that also use DE variants.

The results around the global optimum for this problemare readily available in the literature. For example, Penaet al. (1999) study in detail four of the most popular ini-tialization methods for k-means. Their results show thatany of these four initialization schemes can achieve results

123

Towards improved benchmarking of black-box optimization algorithms using clustering…

Table 5 Previous results for theIris (k = 3, n = 12) problem

References Algorithm fave ( fstd) # f Evals

Maulik and Bandyopadhyay (2000) GA 97.10077 104

Maulik and Bandyopadhyay (2000) KM (best trial of 5) 97.204574

Ye and Chen (2005) KM 98.1872 ?

Ye and Chen (2005) Fuzzy c-means 96.9280 ?

Ye and Chen (2005) AKPSO 96.7551 ?

Kao and Cheng (2006) KM 99.84 104

Kao and Cheng (2006) Shelokar ACO 97.78 104

Kao and Cheng (2006) ACOC 97.22 104

Fathian et al. (2007) HBM 96.95316 11214

Fathian et al. (2007) ACO 97.171546 10998

Fathian et al. (2007) GA 125.197025 38128

Fathian et al. (2007) TS 97.868008 20201

Fathian et al. (2007) SA 97.134625 29103

Xavier (2010) HSC 78.851 (0) ?

Taherdangkoo et al. (2013) SBKM 101.3672 3e04(?)

Taherdangkoo et al. (2013) GAPS 97.3868 3e04(?)

Taherdangkoo et al. (2013) VGAPS 96.2022 3e04(?)

Taherdangkoo et al. (2013) PSO 96.0176 3e04(?)

Taherdangkoo et al. (2013) ACO 99.9176 3e04(?)

Taherdangkoo et al. (2013) SA 101.4574 3e04(?)

Taherdangkoo et al. (2013) BNMR 95.0927 3e04(?)

Hatamlou (2013) KM 105.72902 (12.78359) ?

Hatamlou (2013) PSO 98.14236 (0.84207) ?

Hatamlou (2013) GSA 96.73105 (0.02761) ?

Hatamlou (2013) BB-BC 96.76537 (0.20456) ?

Hatamlou (2013) BH 96.65681 (0.00173) ?

Xiang et al. (2015) DSDE 96.65 (0) 1e04

Xiang et al. (2015) DE/rand/1/bin 96.7 (0.1485) 1e04

Xiang et al. (2015) DE/best/1/bin 101.29 (2.6852) 1e04

Berthier (2015) DE 80.32188 (10) 20023a

Berthier (2015) PDE 78.85212 (0.0015) 21965a

A question mark (?) means that it is not clear from the paper what value was used in the experimentsa An average number of function evaluation evaluations for all successful runs rather than a fixed budget forall runs

close to the global optimum. There are clearly unresolvedquestions here, demonstrating the need for the developmentof standard specifications and experimental practice whenevaluating black-box optimization algorithms. Researchersmaking their code available will also facilitate with repro-ducibility.

7 Conclusion

This paper has examined sum of squares clustering prob-lems as a source of real-world benchmark problems for theevaluation and comparison of black-box optimization algo-rithms. Clustering problems were analysed (Sect. 3) and

found to havemany properties thatmake themhighly suitablefor benchmarking. In particular, they produce a challenging,scalable set of problem instances from a real-world problemdomain and are relatively easy to understand and approachwith unconstrained, continuous black-box optimization algo-rithms. This problem set (and other clustering problems)provide a platform to assist researchers in evaluating the per-formance of the black-box algorithms they have developed.

To facilitate better comparisons of algorithms and exper-imental results, a specification was provided for clusteringproblems and a web repository has been created. Experi-mental results were presented for several commonly usedalgorithms on the set of 27 clustering problems. Overall, theresults show that CMA-ES, a black-box evolutionary algo-

123

M. Gallagher

rithm, provides relatively goodperformance onmanyof theseproblems (with somevariation of population size), often find-ing the global optimum and sometimes far outperforming thestandard Matlab solver (Nelder–Mead) and the widely-used(non black-box) k-means clustering algorithm. It is also clearthat some problems are much more challenging than others:for some of the larger (German Towns) problem instances;there seems to be considerable room for improvement onthe results presented here. Finally, some comparisons weremade with existing results in the literature. The comparisonhighlights the difficulties of comparing experimental results,in particular the lack of a baseline for performance and stan-dards of reporting results.

Clustering is one class of problems, but the field ofcontinuous black-box optimization would benefit from thedefinition of other classes of problems, to build up a morecomprehensive suite of benchmark sets. These problem setsshould complement each other, for example the BBOB prob-lem set contains different kinds of problems compared toclustering. If a variety of problems are then used to evaluatealgorithms in a more standardised way, it should be possibleto improve our understanding of which algorithms work wellon which problem types, which algorithms do not and mostimportantlywhy. The aim is not to produce the best clusteringalgorithm, though it will be interesting to see how black-boxalgorithms compare with clustering algorithms that have theconsiderable advantage of knowledge about the problem.

It is intended that future work will build on and add to theproblems specified here. This can readily be done by consid-ering higher-dimensional problems (by increasing the valueof k), as well as using additional datasets. Exploratory land-scape analysis techniques could be employed to provide someinsights into the structure of the fitness landscapes for theseproblem instances. Categorization of the landscapes willfacilitate more focussed experimental studies, for examplestudying the performance of an algorithm as the number oflocal optima increases. The experimental results in this paperprovide some baselines for performance, however they arelimited in severalways.Additional,well-regarded algorithmsshould be applied, including variation of the parameters ofthese algorithms and how this effects the performance. Also,future research should be able to make better use of pub-lished experimental results on clustering problems. Once wehave a sense of what kind of function evaluation budget isrequired by algorithms to solve a given problem instance,additional algorithms can be more clearly seen as competi-tive or not or certain problem instances. A record of previousresults would hence be of considerable value to the researchcommunity as well as a guide for practitioners seeking toapply algorithms to solving their problem instances. It shouldalso be remembered that clustering problems alone do notrepresent the space of real-world continuous optimizationproblems. More benchmark problems based on other real-

world problem classes would make a valuable contributionto the research community.

Compliance with ethical standards

Conflict of interest The author declares that he has no conflict ofinterest.

References

Berthier V (2015) Progressive differential evolution on clusteringreal world problems. In: Artificial evolution 2015, EA 2015—international conference on artificial evolution. Springer, Lyon.https://hal.inria.fr/hal-01215803

Blake C, Keogh E, Merz C (1998) UCI repository of machine learn-ing databases. Retrieved from http://www.ics.uci.edu/~mlearn/MLRepository.html

Brimberg J, Hansen P, Mladenovic N, Taillard ED (2000) Improve-ments and comparison of heuristics for solving the uncapacitatedmultisource Weber problem. Oper Res 48(3):444–460

Chang DX, Zhang XD, Zheng CW (2009) A genetic algorithm withgene rearrangement for k-means clustering. Pattern Recognit42(7):1210–1222

Du Merle O, Hansen P, Jaumard B, Mladenovic N (2000) An interiorpoint algorithm for minimum sum-of-squares clustering. SIAM JSci Comput 21(4):1485–1505

Fathian M, Amiri B, Maroosi A (2007) Application of honey-beemating optimization algorithm on clustering. Appl Math Comput190(2):1502–1513

Gallagher M (2000) Multi-layer perceptron error surfaces: visualiza-tion, structure andmodelling. PhD thesis,Department ofComputerScience and Electrical Engineering, University of Queensland

GallagherM (2014) Clustering problems formore useful benchmarkingof optimization algorithms. In: Simulated evolution and learning,(SEAL 2014). Springer, pp 131–142

Hatamlou A (2013) Black hole: a new heuristic optimization approachfor data clustering. Inf Sci 222:175–184

Hecht-Nielsen R (1990) Neurocomputing. Addison-Wesley, ReadingHooker JN (1996) Testing heuristics: we have it all wrong. J Heuristics

1:33–42Jain AK, Murty MN, Flynn PJ (1999) Data clustering: a review. ACM

Comput Sur 31(3):264–323Kanade PM, Hall LO (2007) Fuzzy ants and clustering. Syst Man

Cybern Part A: IEEE Trans Syst Hum 37(5):758–769Kao Y, Cheng K (2006) An ACO-based clustering algorithm. In:

Ant colony optimization and swarm intelligence (ANTS 2006).Springer, Berlin, pp 340–347

Likas A, Vlassis N, Verbeek JJ (2003) The global k-means clusteringalgorithm. Pattern Recognit 36(2):451–461

LiuR, Shen Z, Jiao L, ZhangW (2010) Immunodominance based clonalselection clustering algorithm. In: 2010 IEEE Congress on Evolu-tionary Computation (CEC), pp 1–7

Macready W, Wolpert, D (1996) What makes an optimization problemhard? Technical Report. SFI-TR-95-05-046, The Santa Fe Institute

MaulikU,BandyopadhyayS (2000)Genetic algorithm-based clusteringtechnique. Pattern Recognit 33(9):1455–1465

McGeochCC (2002) Experimental analysis of optimization algorithms.In: Pardalos PM, ResendeM (eds) Handbook of applied optimiza-tion, chap 24. Oxford University Press, Oxford, pp 1044–1052

Pena JM, Lozano JA, Larranaga P (1999) An empirical comparisonof four initialization methods for the k-means algorithm. PatternRecognit Lett 20(10):1027–1040

123

Towards improved benchmarking of black-box optimization algorithms using clustering…

Rardin RL, Uzsoy R (2001) Experimental evaluation of heuristic opti-mization algorithms: a tutorial. J Heuristics 7:261–304

Salhi S,GamalMDH (2003)Agenetic algorithmbased approach for theuncapacitated continuous location-allocation problem. Ann OperRes 123:230–222

Shelokar P, JayaramanVK,KulkarniBD (2004)An ant colony approachfor clustering. Anal Chim Acta 509(2):187–195

Steinley D (2006) K-means clustering: a half-century synthesis. Br JMath Stat Psychol 59:1–34

Stephens M (2000) Dealing with label switching in mixture models. JR Stat Soc (B) 62(4):795–809

TaherdangkooM,Hossein ShirzadiM,YazdiM,HadiBagheriM (2013)A robust clusteringmethod based on blind, nakedmole-rats (bnmr)algorithm. Swarm Evolut Comput 10:1–11

Vattani A (2011) k-means requires exponentially many iterations evenin the plane. Discret Comput Geom 45(4):596–616

Wolpert DH, Macready WG (1997) No free lunch theorems for opti-mization. IEEE Trans Evol Comput 1(1):67–82

Xavier AE (2010) The hyperbolic smoothing clusteringmethod. PatternRecognit 43(3):731–737

XiangWL,ZhuN,MaSF,MengXL,AnMQ(2015)Adynamic shuffleddifferential evolution algorithm for data clustering. Neurocomput-ing 158:144–154

Xu R,Wunsch D II (2005) Survey of clustering algorithms. IEEE TransNeural Netw 16(3):645–678

Ye F, Chen CY (2005) Alternative kpso-clustering algorithm. TamkangJ Sci Eng 8(2):165

123