Core instances for testing: A case study

European Journal of Operational Research 166 (2005) 51–62

www.elsevier.com/locate/dsw

Core instances for testing: A case study

Monaldo Mastrolilli *, Leonora Bianchi

IDSIA––Istituto Dalle Molle di Studi sull’Intelligenza Artificiale, Strada Cantonale Galleria 2, CH-6928 Manno, Switzerland

Received 29 October 2002; accepted 30 October 2003Available online 27 July 2004

Abstract

Data generation for computational testing of optimization algorithms is a key topic in experimental algorithmics.Recently, concern has arisen that many published computational experiments are inadequate with respect to theway test instances are generated. In this paper we suggest a new research direction that might be useful to cope withthe possible limitations of data generation. The basic idea is to select a finite set of instances which �represent� the wholeset of instances. We propose a measure of the representativeness of an instance, which we call e-representativeness: for aminimization problem, an instance xe is e-representative of another instance x if a (1+ e)-approximate solution to x canbe obtained by solving xe. Focusing on a strongly NP-hard single machine scheduling problem, we show how to mapthe infinite set of all instances into a finite set of e-representative core instances. We propose to use this finite set ofe-representative core instances to test heuristics.� 2004 Elsevier B.V. All rights reserved.

Keywords: Testing methodology; Experimental analysis of algorithms; Approximation algorithms; Scheduling; Optimization problems

1. Introduction

1.1. Motivation

The literature addressing computational testingof optimization algorithms has a long history and,until recently, computational testing has been per-formed primarily through empirical studies. Inthe operations research community, concern has

0377-2217/$ - see front matter � 2004 Elsevier B.V. All rights reservdoi:10.1016/j.ejor.2003.10.057

* Corresponding author. Tel.: +41 91 610 86 64; fax: +41 91610 86 61.

E-mail address: [email protected] (M. Mastrolilli).

arisen that many published computational experi-ments are inadequate, especially with respect tothe way test instances are generated (see[1,9,12,13,15,19]). Among the limitations of com-putational tests is that they often involve simplydevising a set of supposedly �typical� instances,running the algorithm and its competitors onthem, and comparing the results. The usefulnessand importance of such experiments strongly de-pends on how �typical� the sample instancesactually are. Moreover, the number of such com-parisons is quite limited. As a consequence, itis probably unwise to make any hard and fast

ed.

mailto:[email protected]

52 M. Mastrolilli, L. Bianchi / European Journal of Operational Research 166 (2005) 51–62

generalizations by using this kind of computa-tional tests.

1.2. Our contribution

In this paper we suggest a new research direc-tion that might be useful to cope with the possiblelimitations of data generation. The basic idea is toselect a finite set of typical instances which �repre-sent� the whole set of instances. The key phrasehere is �represent�. We concentrate on the conceptof representativeness of a set of problem instances.We define a measure of the representativeness ofan instance that we call e-representativeness. Givena minimization problem 1 P, an instance xe is e-representative of another instance x if a (1+d)-approximate solution for xe can be transformedin polynomial time into a (1+d+ e)-approximatesolution for x, for e, dP 0. In other words, if a�good� solution for xe can be mapped into a �good�solution for x.

In this paper we restrict to a strongly NP-hardsingle machine scheduling problem, which in thenotation of Graham et al. [8], is noted 1jrj, qjjCmax.For this problem we propose a methodology forexperimental testing of algorithms which is basedon the concept of e-representativeness. The testmethodology works as follows. For any e > 0, wemap the infinite set of all instances I into a finiteset of instances Ie which are e-representative ofthe instances in I. We call Ie core set, and the in-stances belonging to it core instances. More pre-cisely, we exhibit a linear time algorithm f thattransforms any given instance x from I into an in-stance xe that is e-representative of instance x, andthat belongs to a defined finite set Ie. Moreover, a(1+d)-approximate solution for xe can be trans-formed in linear time (by algorithm g) into a(1+d+ e)-approximate solution for x, for anydP 0. Assume now that you have a heuristic He

that has been tested on the finite set of coreinstances Ie. If He is an (experimentally) good heu-ristic He (i.e. an algorithm that returns (1+d)-approximate solutions) for this finite set Ie of core

1 A similar definition can be given for maximization prob-lems.

instances, then it can be used to get a good algo-rithm for the infinite instances of problem1jrj, qjjCmax (i.e. (1+d+ e)-approximate algo-rithm). In Fig. 1 we give a pictorial view of the(1+d+ e)-approximation algorithm that works asfollows: for any given instance x from I, first trans-form x into xe, then apply He on xe, and transformback the obtained solution for xe into (1+d+ e)-approximate solution for x. It follows that by test-ing the heuristic on the core instances, we canobtain an approximation algorithm whose per-formance ratio depends on the experimentalresults and on the representativeness e of theinstances used for testing. We remark that thisapproach can be seen as a �black-box� testing andused to test any kind of heuristics (such as simu-lated annealing, genetic algorithms, tabu search,ant colony optimization algorithms, etc.).

Unfortunately, we could only find a core in-stance set whose cardinality is exponential in 1/e.A natural question addressed in this paper isto understand if we can hope to reduce considera-bly the size of the core set, i.e., to be polynomial in1/e. For strongly NP-hard problems, we exhibit afirst negative results which states that the size ofIe cannot be a polynomial in 1/e, unless P=NP.(We do not know if weakly NP-hard problems ad-mit a polynomial in 1/e core set size.) This meansthat for �small� values of e the number of core in-stances may actually be too large to be fully tested.This limitation suggests a second natural question,that is, what can we say about the performances ofan algorithm by performing a �small� (polynomialin 1/e) number of tests? A first positive answer tothis question can be obtained by using Monte

Fig. 1. The resulting (1+d+ e)-approximation algorithm.

M. Mastrolilli, L. Bianchi / European Journal of Operational Research 166 (2005) 51–62 53

Carlo method for sampling on the core set. Weshow that within a �small� number of tests we cancheck if an algorithm performs �well� (or betterthan another algorithm) for most of the core in-stances with high probability.

We believe that this paper presents a new re-search direction and shows only preliminary re-sults. Many questions remain to be explored tounderstand if it can be applied in a direct wayfor experimental analysis of algorithms/heuristics.

1.3. Techniques

Interestingly, the techniques used to reduce I toIe are a combination of the �standard� techniques toobtain polynomial time approximation schemes(PTAS) (see [21] for a survey). In the literaturethere are several polynomial time approximationschemes that make use of the relative error e to re-duce the number of potentially �interesting� solu-tions from exponential to polynomial. We exploitthe error e to reduce the number of �interesting� in-stances. These strong similarities let us hope thatanalogous results can be obtained for several otherproblems that admit a PTAS. Indeed, we claimthat, by using the ideas introduced in this paperand the techniques described in [7], the presentedapproach for testing can also be used for otherscheduling problems, such as the identical andunrelated parallel machine scheduling problemwith costs, the flow-shop and the job-shop sched-uling problems with fixed number of machinesand of operations per job. More generally, we be-lieve that this approach can be extended to severalother problems belonging to the class EPTAS. 2

1.4. Organization of the paper

The remainder of the paper is organized as fol-lows. Section 2 focuses on the case study of the sin-gle machine scheduling problem, for which, inSection 2.1 we derive the core instance set. Section3 shows the theoretical lower bound on the core set

2 EPTAS is the class of optimization problems which admitapproximation schemes running in time f(1/e)nc, where f is anarbitrary function, n is the input size and c is a constantindependent of e (see [4]).

cardinality. Section 4 shows how to sample andhow to test a subset of core instances by meansof Monte Carlo sampling. In Section 5 we applythe principles of previous chapters to the experi-mental evaluation of two Branch & Bound(B&B) algorithms. Conclusions an future researchissues are in Section 6.

2. Case study: Scheduling on a single machine

We shall study the following problem. There isa set of n jobs Jj (j=1, . . .,n). Each job Jj must beprocessed without interruption for pjP 0 timeunits on a single machine, which can process atmost one job at a time. Each job has a release daterjP 0, which is the time when it first becomesavailable for processing. After completing itsprocessing on the machine, a job requires an addi-tional delivery time qjP 0. If sj (P rj) denotes thetime Jj starts processing, job Jj is delivered at timeCj= sj + pj + qj. Delivery is a non-bottleneck activ-ity, in that all jobs may be simultaneously deliv-ered. Our objective is to minimize, over allpossible schedules, the maximum delivery time,i.e., Cmax =maxjCj. The problem as stated isstrongly NP-hard [16], and in the notation ofGraham et al. [8] it is noted as 1jrj,qjjCmax.

In this paper we want to use the test methodologybased on e-representativeness for the experimentaltest of two exact algorithms, the general ECLiPSesolver and the B&B algorithm by Carlier [3].

A key feature of Carlier�s B&B is the O(n logn)heuristic which schedules jobs using the general-ized earliest due date of Jackson [14]. Specifically,this heuristic selects, from the available jobs, onewith the largest delivery time to schedule next.Carlier�s algorithm appears to be the most efficientone for the 1jrj, qjjCmax problem. We implementedCarlier�s B&B in C++, and it is available fordownloading at http://www.idsia.ch/~leo/.

ECLiPSe (ECLiPSe Common Logic Program-ming System) is a Prolog based system whoseaim is to serve as a platform for integratingvarious Logic Programming extensions, in particu-lar Constraint Logic Programming. The kernelof ECLiPSe is an efficient implementation ofstandard (Edinburgh-like) Prolog as described in

http://www.idsia.ch/~leo/


basic Prolog texts [5]. ECLiPSe is a software sys-tem for the cost-effective development and deploy-ment of constraint programming applications, e.g.in the areas of planning, scheduling, resource allo-cation, timetabling, transport etc. ECLiPSe is in-tended for general programming tasks, especiallyrapid prototyping. Up-to-date information canbe obtained from the ECLiPSe web site (http://www.icparc.ic.ac.uk/eclipse). We have formalizedproblem 1jrj,qjjCmax in ECLiPSe (available athttp://www.idsia.ch/~leo/) and the solution isfound using a built-in B&B method.

The first phase of our test method requires map-ping the set of all instances to the set of core in-stances. This is performed in Section 2.1.

2.1. Core reduction

We start defining set Ie of core instances, thenwe prove that the infinite set I of all instancescan be mapped to the finite set Ie, for every e > 0.In the rest of the paper we assume, for simplicityof notation, that 1/e is an integer. Letm ¼ ð1 þ 1

e Þ2 þ 2

e ¼ Oð1=e2Þ. The set of integers0,1, . . .,k is denoted [k].

Definition 1. An instance xe belongs to Ie if andonly if the following holds:

1. xe has m jobs;2. for every job j of xe, rj=h Æ e and qj= i Æ e for someh, i2 [1/e];

3. for every job j of xe, pj ¼ i em for some i2 [m/e].

The size of Ie is bounded by (1/e)O(1/e2). Indeed,let us say that two jobs are of the same type if theyhave the same processing, release and delivery time.Clearly, the number of different types is boundedby s=O(1/e5). Let us denote these s types byt1, . . ., ts. Let ui denote the total number of jobs oftype ti, for i=1, . . .,s. A core instance is any vector(u1,u2, . . .,us) with

Pui ¼ m. Therefore, we have at

most sm=(1/e)O(1/e2) core instances.The following theorem proves that the infinite

set I of all instances can be mapped to Ie. Our ap-proach uses several transformations of the giveninput instance x which may potentially increase

the objective function value by a factor of1+O(e). Therefore we can perform a constantnumber of transformations while still staying with-in 1+O(e) of the original optimum. At the end, theresulting transformed instance xe belongs to Ie andany (1+d)-approximate solution for xe can betransformed in linear time into a (1+d+O(e))-approximate solution for x. Throughout thispaper, when we describe this type of transforma-tion, we shall say it produces 1+O(e) loss. (Theproof of the following theorem can be found inAppendix.)

Theorem 1. For any fixed e > 0, with 1+6e loss,

any given instance x2 I can be transformed into an

instance xe2 Ie in O(n) time, where the constant

hidden in the O(n) running time is reasonably small

and does not depend on the error e.

Note that, although the size of Ie may be de-creased, we prove in Section 3 that jIej cannot bepolynomial in 1/e, unless P=NP.

3. On the core set size

In 1977, Berman and Hartmanis [2] investigatedthe density of NP-complete problems. The densityof a language A is a function CA : N ! N definedby CA= j{x2A:jxj6 n}j. Recall that a language Ais sparse if there exists a polynomial q such that forevery n 2 N, CA(n)6 q(n). In [2], Berman andHartmanis proposed the density conjecture, thatno sparse language can be NP-complete. The den-sity conjecture was proved to be equivalent toP 6¼NP by Mahaney [17].

In this section we give a general condition thatassures that a problem P cannot have a core setsize which is polynomial also in 1/e. Theorem 2 ap-plies to the problem addressed in this paper. Theproof of the following theorem uses the resultproved by Mahaney [17].

For any problem P and for any instance x of P,let max(x) denote the value of the largest numberincurring in x2 I, where I denotes the set of in-stances of problem P. For simplicity, let us focuson minimization problems. Let SOL(x) be a func-tion that associates to any input instance x2 I the

http://www.icparc.ic.ac.uk/eclipse

http://www.icparc.ic.ac.uk/eclipse

http://www.idsia.ch/~leo/


set of feasible solutions of x. Let m be a functionthat associates with any instance x2 I and withany feasible solution y2 SOL(x) a positive rationalm(x,y) that denotes the measure of solution y. Thevalue of any optimal solution of x will be denotedas m(x). The following definition formalize thenotion of core reducibility, that maps instancesof I to instances of Ie, but it also maps backgood solutions for Ie to good solutions for I.

Definition 2. For any fixed e > 0, problem P is core

reducible, if two functions f and g, and a finite set Ie

of instances exist such that, for any instance x2 I:

1. f(x, e)2 Ie;2. for any y2 SOL(f(x, e)), g(x, y, e)2 SOL(x);3. f and g are computable in polynomial time with

respect to both the length of the instance and 1/e;4. there exists a polynomial time computable solu-

tion s2 SOL(f(x, e)) such that m(x, g(x, s, e))6(1+ e) Æm(x).

We are now ready to give the following funda-mental result. (The proof of the following theoremcan be found in Appendix.)

Theorem 2. Let P be a strongly NP-hard problemthat admits a polynomial p such that m(x)6

p(jxj,max(x)), for every input x2 I. If P is core

reducible, then the density of Ie cannot be polyno-

mial in 1/e, for any e > 0, unless P=NP.

The two main requirements of Theorem 2 arethat P have solution values that are not too largeand that P is NP-complete in the strong sense.These hold for many problems of interest, includ-ing, for example, the bin packing problem, thegraph coloring problem, the maximum independ-ent set problem and the minimum vertex coverproblem. Theorem 2 also applies to the addressedscheduling problem. Thus we have a generalmethod of considerable power for ruling out thepossibility of a core set size polynomial in 1/e.

4. Sampling on the core set

In Section 2.1 we have presented a core set forproblem 1jrj,qjjCmax whose cardinality is exponen-

tial in 1/e. Moreover, in Section 3 we have shownthat it is unlikely that a �small� core set exists.Thus, for �small� values of the e parameter, the coreset may be too big to be fully tested. A way to dealwith this problem is to select a subset of the coreset which is small enough to be fully tested. In thiscase the problem is to select an appropriate subsetfor which any measurement done on the subset is agood estimator of a measurement on the wholecore set. In this section we use Monte Carlo sam-pling to select by random sampling an appropriatesubset of the core set, in order to do algorithmtesting. Then we show how to apply this methodto the comparison of two algorithms.

4.1. Testing by Monte Carlo sampling

Let P be a core reducible problem and Ie be thecore set of this problem for every fixed e > 0. Let Abe an algorithm for P which we want to evaluateby experimental testing. Let us define a test tomake on A, that is, let us define a Boolean functionf : I e ! f0; 1g over Ie. We define the set I1

e ¼fx 2 I e j f ðxÞ ¼ 1g as the pre-image of 1. The func-tion f may be thought of as an indicator of success(f(x)=1) or failure (f(x)=0) of the algorithm A

over the core set Ie. For example, given a core in-stance x2 Ie, the algorithm A has success when itfinds a solution which is within a certain distancefrom the optimum, while A fails in the oppositecase. The problem is to estimate the size of I1

e withrespect to the size of Ie, that is, the fraction j I1

e j =j I e j of �successes� in our test of the algorithm.

An obvious approach to estimating j I1e j = j I e j

is to use the classical Monte Carlo method (see,e.g. [20]). This involves choosing N independentsamples from Ie, say x1,. . ..,xN, and using the valueof f on these samples to estimate the probabilitythat a random choice will lie in I1

e . More formally,define the random variables X1, . . .,XN as follows:

X i ¼1 if f ðxiÞ ¼ 1;

0 otherwise:

�

By this definition, Xi=1 if and only if xi 2 I1e .

Finally, we define the estimator random variable

Z ¼j I e jXNi¼1

X i

N:


It is easy to verify that E½Z ¼j I1e j. We might

hope that with high probability the value of Z isa good approximation to j I1

e j. In particular, wewant to have a small error with respect to jIej,when using Z as an estimator for j I1

e j. Of course,the probability that the approximation is good de-pends upon the choice of N. The following theo-rem (see also [20]) relates the value of N to g and d.

Theorem 3. For g2 (0,1] and d2 (0,1], the MonteCarlo method computes an estimator Z such that

Prj I1

e jj I e j

� Zj I e j

�� < g

� �P 1 � d;

provided

N P4

g2ln

2

d:

In practice, after running A on the sample, wecount the number of times A has been successful.Let X denote this value (X is linked to the estima-tor Z by Z ¼j I e j X

N). Theorem 3 implies that, pro-vided the sample is big enough, j I1

e j lies in theinterval ððXN � gÞ j I e j; ðXN þ gÞ j I e jÞ, with probabil-ity at least 1�d. Therefore, with probability atleast 1�d, algorithm A is successful for at leastthe ðXN � gÞ � 100% of core instances.

4.2. Comparison of two algorithms

The goal of this section is to describe a methodto compare algorithms. Let P be a core reducibleproblem and Ie be the core set of this problemfor every fixed e > 0. Let A and B be two algo-rithms for P which we want to compare. Let us de-fine a test to make on A and B, that is, let us statewhen an algorithm is �better�, �worse� or �equiva-lent� to another algorithm. For example, given acore instance x2 Ie, the algorithm A is better thanB on x if it returns a better solution than B.

In the following we are interested into comput-ing how many times A is better, worse or equiva-lent to B. For every given algorithm A and B theset Ie can be seen as partitioned into three subsetsIAe ; I

Be and IA¼B

e of instances, where IAe is the set ofinstances for which A is better than B, IBe the oppo-site and IA¼B

e is the set of instances for which A andB are equivalent. We are interested in the size of

these subsets and in the following we describehow to approximate these values.

Let g and d be two parameters, and let N be thesize of a sample from the core set. After running Aand B on the same sample, we count the number oftimes A is better, worse or equal to B. Let a, b andc denote, respectively, these values. Then, we de-fine ZA

e ¼j I e j aN, ZB

e ¼j I e j bN and ZA¼B

e ¼j I e j cN as

the estimators of j IAe j, j IBe j and j IA¼Be j, respec-

tively. Again it is easy to verify that E½ZAe ¼j IAe j,

E½ZBe ¼j IBe j and E½ZA¼B

e ¼j IA¼Be j. By Theorem

3, we can say that, provided the sample is big en-ough (i.e. N P 4

g2 ln 2d), j IAe j, j IBe j and j IA¼B

e j lie

in intervals ððaN � gÞ j I e j, ða

N þ gÞ j I e jÞ, ððbN � gÞ

j I e j, ðbN þ gÞ j I e jÞ and ðð c

N � gÞ j I e j, ð cN þ gÞ

j I e jÞ, respectively, with probability at least 1�d.

4.3. Sampling from the core set of 1jrj, qjjCmax

The methods described in Sections 4.1 and 4.2can be applied to problem 1jrj, qjjCmax if we canchoose, uniformly at random, N independent sam-ples from the core set Ie of Definition 1. This canbe done without generating the (exponentiallylarge) core set Ie. Indeed, we can choose, uniformlyat random, an instance from Ie as follows. Recallthat s=O(1/e5) is the number of different jobtypes, and that m is the number of jobs of any in-stance from Ie (see Section 2.1). Choose independ-ently and uniformly at random m numbers from [s].Each number defines the type of the m jobs ofthe instance. It is easy to check that any instancefrom Ie has the same probability to be selected.The number of random bits used is polynomialin 1/e.

We observe that every polynomial algorithm forproblem 1jrj, qjjCmax can be tested in time polyno-mial in 1/g, log(1/d) and 1/e. Indeed, the requirednumber of experiments is a polynomial functionof 1/g and log (1/d), and the size of every core in-stance is polynomial in 1/e.

5. Experimental tests

In this section we provide two applications ofthe described testing methodology. We remark

8000

10000

12000

14000

16000

18000

20000

22000

24000

0 20 40 60 80 100

perf

orm

ance

rat

io

percentage of core instances

Fig. 2. Comparison between Carlier�s B&B and ECLiPSe B&B.Performance ratio is the ratio of runtimes between ECLiPSeB&B and Carlier�s B&B. Percentage is the percentage of coreinstances where the corresponding performance ratio isobserved, with probability 0.9.


that this approach can be seen as a �black-box�testing and used to test any kind of algorithm(such as simulated annealing, genetic algorithms,tabu search, etc.).

5.1. Comparison of branch & bound algorithms

The goal of this section is to compare an ad-hocB&B method (Carlier�s B&B) with the general sol-ver offered by ECLiPSe. We would like to under-stand what is the trade-off between the rapidprototyping offered by ECLiPSe and the effective-ness of Carlier�s B&B (which requires much moretime to be implemented).

Since exact algorithms always return the opti-mal solution value, here the goal is to evaluatethe time required to compute it. In particular, weanalyze how many times Carlier�s B&B is fasterthan ECLiPSe B&B. We assume that the test issatisfactory when d=0.1, g=0.1 and e=0.2. ByTheorem 3 it suffices to consider 1200 ð> 4

g2 ln 2dÞ

core instances chosen uniformly at random. Weperformed the test on a Pentium 1500 MHz with260 MB. We observed that Carlier�s B&B foundthe optimal solution for every instance in less than0.02 seconds, whereas after 150 seconds ECLiPSeB&B never found the optimal value for none ofthe samples. More precisely, ECLiPSe B&B tookat least 172 and at most 1879 seconds to find theoptimal solutions of the test instances. Now we ap-ply the ideas of Section 4.2 with the goal to under-stand on which fraction of instances Carlier�s B&Bis at least 8500 (=172/0.02) times faster thanECLiPSe B&B. We conclude that with probability1�d=0.9 Carlier�s B&B is at least 8500 faster thanECLiPSe B&B for the 90% (=(1�g)100) of thecore instances with e=0.2. Other results are re-ported in Fig. 2, where it is shown the ratio of run-times between ECLiPSe B&B and Carlier�s B&B.This ratio is displayed as a function of the percent-age of instances for which this ratio holds withprobability 0.9. (For example in Fig. 2 we observethat on the 50% of core instances Carlier�s B&Bis 16000 times faster than ECLiPSe B&B withprobability 0.9.)

The evident superiority of Carlier�s B&B de-serves further investigations which are providedin Section 5.2.

5.2. Evaluation of Carlier’s branch & bound

We tested the B&B by Carlier for three differentcore instance sets, corresponding to e=0.005,e=0.01, e=0.1. The first two tests (e=0.005,0.01) were performed on a COMPAQ AlphaServerDS20E 666 MHz, which uses 64-bit LONG INTtype, necessary for the evaluation of the objectivefunction. The third test (e=0.1), was performedon an Intel(R) Xeon(TM) CPU 1.70 GHz ma-chine, which uses 32-bit LONG INT type.

The aim of our test was to verify the actualcomplexity of the B&B, in terms of the numberof nodes expanded by the branch and bound algo-rithm. We found that for all the tested core in-stances, Carlier�s B&B only expands one node ofthe search tree, showing that the initial solutionprovided by the Jackson-rule [14] is already opti-mal when the instances are generated uniformly.This fact has been already observed (experimen-tally) several times for other uniformly selectedinstances (see e.g. [3]). Results are reported inTable 1.

We conclude by noting that the described ex-perimental results are confirmed by the followingtheorem (proof in Appendix).

Theorem 4. If the sampled instances are independ-

ently and uniformly distributed in Ie then the solution

Table 1Experimental results of Carlier�s B&B, tested over three different core sets Ie = 0.005, Ie = 0.01, Ie = 0.1

e N X/N d g PC1 (%) Pr{PC1}

0.005 13,789 1 10�5 3.5·10�3 99.65 0.999990.01 170,267 1 10�5 2.9·10�4 99.971 0.999990.1 11,630,000 1 10�5 4.2·10�6 99.99958 0.99999

N is the number of sampled core instances, X is the number of core instances where the B&B only expanded one node to find theoptimum, d and g are the Monte Carlo method parameters. The percentage of core instances where the B&B only needs one node is atleast PC1, with probability Pr{PC1}.


value mJ(m) returned by Jackson rule satisfies the

following inequality:

PrfmJðmÞ=mðmÞP 1 þ 4=mg6 e�m=8;

where m(m) denotes the optimal value and m=

O(1/e2).

6. Future research

Focusing on a strongly NP-hard single machinescheduling problem, we have shown how to mapthe infinite set of all instances into a finite set ofe-representative core instances. The concept of e-representativeness has been applied to test andcompare two Branch & Bound algorithms. We be-lieve that this paper is just an initial step in theexploration of such idea for testing and so we stateseveral open problems and possible directions. Wehave seen that the number of core instances cannotbe polynomial in 1/e when the problem is stronglyNP-hard. Does the latter hold also for a weaklyNP-hard problem? We observed, by sampling uni-formly, that Carlier�s B&B performs �well� for mostof the core instances. Other ways of samplingshould be investigated to understand on which in-stances this algorithm performs poorly. Finally, webelieve that there are possibilities to connect thiswork to smoothed analysis of algorithms [22]and parameterized complexity [6].

Acknowledgements

This research has been supported by the SwissNational Science Foundation project 200021-

100539/1, ‘‘Approximation Algorithms for Ma-chine scheduling Through Theory and Experi-ments’’, and by the ‘‘Metaheuristics Network’’,grant HPRN-CT-1999-00106.

Appendix A. Proof of Theorem 1

We start describing procedure Core-Reductionwhich maps the infinite set I of all instances to afinite set Ie of core instances. Then, we analyze thisalgorithm and show that by applying Core-Reduc-tion, any given instance x from I can be trans-formed with (1+6e) loss into a core instancebelonging to Ie, for every e > 0.

The procedure Core-Reduction consists of scal-ing the input data and �merging� jobs to form anapproximate version of the original instance thatbelongs to a finite set of different instances. Theprocedure Core-Reduction is as follows.

1. Checking number of jobs. Let m ¼ ð1 þ 1e Þ

2þ2e ¼ Oð1=e2Þ. If the input instance has no morethan m jobs, go to point (3), otherwise go topoint (2).

2. Grouping small jobs together. Let P ¼P

jpj,rmax =maxjrj and qmax =maxjqj. Round downevery release date (delivery time) to the nearestvalue among the following q ¼ 1þ 1

e ¼ Oð1=eÞvalues: 0, ermax,2ermax, . . ., rmax (0, eqmax,2eqmax,

. . .,qmax). For simplicity of notation, let us userj and qj to denote the resulting rounded releaseand delivery times of any job j. Divide jobs intolarge and small jobs, according to the length oftheir processing times, L={j:pj> eP} andS={j:pj6 eP}. Let us say that two small jobs,


j and h, belong to the same class iff rj= rh andqj=qh. The number of distinct job classes isbounded by j ¼ ð1 þ 1

e Þ2 ¼ Oð1=e2Þ. Let us par-

tition the set of small jobs into at most j classesK1, K2, . . .,Kj. Let j and h be two small jobsfrom Ki such that pj+ph6 eP. We �glue�together these two jobs to form a composedjob with the same release date and delivery timeas j (and h), and in which the processing time isequal to the sum of their processing times. Werepeat this process until at most one job fromKi has processing time not greater than eP/2.At the end, all jobs in class Ki, have processingtimes not greater than eP. The same procedureis performed for every class. At the end of theprocess, the total number of composed jobs isno larger than P

eP=2þ j ¼ m.

3. Normalizing. Let pmax, rmax and qmax denote themaximum processing, release and delivery time,respectively, of the instance after steps (1) and(2). Let LB=max{pmax, qmax, rmax}. Normalizethe instance by dividing every release, deliveryand processing time by LB.

4. Rounding release and delivery times. Rounddown every release date and delivery time tothe nearest value among the followingq=O(1/e) values: 0, e, 2e, . . ., 1.

5. Rounding processing times. Round down everyprocessing time to the nearest value amongthe following p=O(1/e3) values: 0, e/m,2e/m, . . ., 1.

In the following we analyze Core-Reductionand show that the transformation is with (1+6e)loss. First we observe that if the number of jobsof the input instance is not less than m then, by fol-lowing the ideas in [18], small jobs can be mergedtogether as described in step (2) with 1+3e loss(see Lemma 1 for a proof of this).

Note that step (3) can be performed with noloss, the normalized instance is equivalent to theone after steps (1) and (2). Observe that after step(3) every processing, release and delivery time isnot greater than 1. Now, let us consider the result-ing instance x after steps (1)–(3), and instance x 0

obtained from x by rounding release, processing,and delivery times as in steps (4) and (5) of

Core-Reduction. In steps (4) and (5), by roundingdown the values we cannot obtain a modified in-stance with an optimal value larger than the opti-mal value of instance x. Following the ideas in[10], every feasible solution for the modified in-stance x 0 can be transformed into a feasible solu-tion for the original instance just by adding e toeach job�s starting time, and re-introducing theoriginal delivery and processing times. It is easyto see that the solution value may increase by atmost 3e (observe that there are at most m jobs andeach processing time may increase by at most e/m).

Let us now focus on the time complexity ofCore-Reduction. It is easy to check that steps(1), (3)–(5) of the procedure requires linear time.Suppose now that step (2) must be performed, thatis, nP m. The process of partitioning small jobsinto j job classes, as described in step (2) of theprocedure, can be implemented to take O(n+j)time. Since in this case nP j, step (2) also takesO(n) time. h

Lemma 1. With 1+3e loss, the number of jobs can

be reduced to be at most min{n,O(1/e2)}.

Proof. In the following we show that by using theinstance with grouped small jobs (as described instep (2) of Core-Reduction) we can get an optimalsolution with 1+3e loss.

Following the ideas in [10], every feasible solu-tion for the modified instance with rounded releaseand delivery times can be transformed into a feasi-ble solution for the original instance just by addingermax to each job�s starting time, and re-introduc-ing the original delivery times. It is easy to see thatthe solution value may increase by at moste(rmax +qmax)6 2eopt, where opt is the optimal va-lue of the original instance (see [10]). Let us con-sider an optimal schedule y for the instance x

with rounded release and delivery times. Modifyx to obtain instance x in which the processingtimes of all jobs remain unchanged, while releaseand delivery times are set as follows:

~rj ¼ rj; ~qj ¼ qj for j 2 S;~rj ¼ sj ; ~qj ¼ optðxÞ � pj � sj for j 2 L;

ðA:1Þ


where sj denote the starting time of job j in y.Clearly opt(x)=opt(x).

Merge small jobs as described before and de-note by ~x the resulting modified instance. By show-ing that there exists a schedule ~y for ~x such thatmð~x; ~yÞ6 ð1 þ eÞoptðxÞ the claim follows.

Consider the following algorithm, known asExtended Jackson�s Rule: whenever the machineis free and one of more jobs is available forprocessing, schedule an available job with largestdelivery time. Apply the Extended Jackson�s Ruleto the modified instance ~x. Let use ~y to denotethe resulting schedule. Let us define a critical jobJc as one that finishes last in y, i.e., its delivery iscompleted last. Associated with a critical job Jc

there is a critical sequence consisting of those jobstracing backward from Jc to the first idle time inthe schedule. Let us fix a critical job Jc and denotethe last job Jb in the critical sequence with ~qc > ~qb

as interference job for the critical sequence. Let Bdenote the set of jobs processed after Jb in the crit-ical sequence. By the way that Jb was chosen,clearly ~qj P ~qc for all jobs Jj2B. We claim thatif there is no interference job then mð~x; ~yÞ ¼optðxÞ, otherwise mð~x; ~yÞ6 optðxÞ þ pb. It is easyto show that

mð~x; ~yÞ6 pb þ minj2B~rj þXj2B

pj þ minj2B~qj

and

optðxÞP minj2B~rj þXj2B

pj þ minj2B~qj:

By the way that the interference job Jb was chosen,we have ~qc ¼ min j 2 B~qj. Let U denote the set Bof jobs obtained by ‘‘unsticking’’ small jobs. Wecan bound the length of an optimal schedule forx as

optðxÞP minj2U~rj þXj2U

pj þ minj2U~qj:

It is easy to see that

minj2B~rj þXj2B

pj þ minj2B~qj

¼ minj2U~rj þXj2U

pj þ minj2U~qj:

Indeed, we have glued only small jobs having thesame release and delivery times, and thereforeminj2B~qj ¼ minj2U~qj and minj2B~rj ¼ minj2U~rj.Therefore, m(~x,~y)6 pb +opt(x) and if there is nointerference job then m(~x,~y)=opt(x).

Then if there is no interference job or Jb is asmall job we have m(~x,~y)6 (1+ e) Æopt(x), by def-inition of small jobs (‘‘glued’’ or not). Now, con-sider the last case in which the interference jobJb is a large job. By the definition of the extendedJackson�s rule, no job j2B could have been avail-able at the time when Jb is processed, since other-wise such a job j would have taken priority overjob Jb. Thus, ~rc > ~rb and ~qc > ~qb, but Jb cannotbe a large job since by construction if ~rc > ~rb then~qb P ~qc. h

Appendix B. Proof of Theorem 2

Let us assume that there exists a polynomial q1

such that the density CIe of Ie is bounded byCIe(n)6 q1(n, 1/e), for every n 2 N and for anye > 0. Since P is strongly NP-hard, there exists apolynomial q2 such that problem ~P obtained byrestricting P to only those instances x for whichmax(x)6 q2(jxj), is NP-hard. We show that theassumption is equivalent to P=NP by reducingproblem ~P to a problem P0 whose underlyinglanguage is sparse and NP-complete.

For every (x, y)2 I·SOL, the measure functionm(x,y) of P is defined to have values in Q. It ishowever possible to transform any such optimiza-tion problem into an equivalent one such thatm(x, y) is a positive integer. We assume this inthe following.

Consider set I(n)={x2 I:jxj=n}, for any n 2 N.For every x2 I(n), if we set the error to e(n)=1/(p(n, q2(n))+1), then any y 2 SOL(f(x, e(n)) canbe transformed into the optimal solution for x

by means of function g. Indeed, since ~P is corereducible we have m(x, g(x, y, e(n)))6 (1+ e(n)) Æm(x) <m(x)+1, where the last inequality is dueto the fact that m(x)6 p(jxj, max(x))6 p(n, q2(n)).From the integrality constraint on the measurefunction m, it follows that m(x, g(x, y, e(n)))=m(x), that is, g(x,y, e(n)) is an optimal solutionfor x. The time complexity to compute f(x, e(n))


and g(x,y, e(n)) is polynomial in jxj. Let us parti-tion set I(n) into equivalence classes, such that x,z2 I(n) belong to the same class iff f(x, e(n))=f(z, e(n)). Consider set I 0(n) which contains for everyequivalence class the corresponding core instance,i.e., I 0(n)={f(x, e(n)):x2 I(n)}. Let q3 be the poly-nomial limiting the computation time of f(x, e(n)),for x2 I(n). Then, for any x2 I(n), jf(x, e(n))j6 q3(n). If jf(x, e(n))j<n we increase the length off(x, e(n)) up to n, such that the modified instanceis equivalent, except for the length, to the former(for instance, we can do that by adding a dummysymbol several times till length n is touched). Forsimplicity, let us denote this modified length-in-creased set by I 0(n) again. It is easy to see thatI(n) polynomially reduces to I 0(n), since a polyno-mial time algorithm for I 0(n) implies the existenceof a polynomial time algorithm for I(n). Accordingto the assumption, the cardinality of I 0(n) isbounded by a polynomial q4(n), where q4(n)=q1(n,p(n,q2(n))+1). Since for every n 2 N, I(n)polynomially reduces to I 0(n), then I ¼ [n2NIðnÞpolynomially reduces to I 0 ¼[n2NI 0ðnÞ, and the den-sity CI0 of I 0 is bounded by a polynomial, indeedCI 0 ðnÞ¼jfx2I 0 :jx j6ngj6 j[i6nI 0ðiÞ j¼

Pi6n jI 0ðiÞ j.

Hence, problem ~P can be reduced in polynomialtime to a problem P0 with a sparse set of instances.It is easy to see that P0 is NP-hard, and the under-lying language is sparse and NP-complete. Thisis equivalent to P=NP [17]. h

Appendix C. Proof of Theorem 3

The claim follows by proving that

Pr½Z6 j I1e j �g j I e j 6

d2

ðA:2Þand

Pr½Z P j I1e j þg j I e j 6

d2; ðA:3Þ

for N P 4g2 ln 2

d.Define X ¼

PNi¼1X i, q ¼j I1

e j = j I e j and theestimator Z= jIejX/N. Then,

Pr½Z6 j I1e j �g j I e j ¼ Pr X 6N

j I1e j

j I e j� Ng

� �

¼ Pr X 6Nq 1 � gq

� �� :

Since E[X]=Nq, by a straightforward applica-tion of the Chernoff bound (see, e.g. [20]) to therightmost term we obtain

Pr½Z6 j I1e j �g j I e j 6 e�Ng2

4q 6 e�Ng2

4 :

For the given lower bound on N we obtain(A.2). Similarly, we can prove (A.3).

It is easy to see that inequalities (A.2) and (A.3)imply

Prj I1

e jj I e j

� Zj I e j

�� < g

� �P 1 � d: �

Appendix D. Proof of Theorem 4

Any instance that is independently and uni-formly distributed in Ie is an instance withm ¼ ð1 þ 1

e Þ2 þ 2

e jobs where processing times areuniformly distributed in f0; e

m ; 2em ; . . . ; 1g, whereas

release and delivery times are uniformly distrib-uted in 0, e, 2e, . . ., 1. Without loss of generality,let us multiply every value by b ¼ m

e, this transformsany instance into another having only integral val-ues. An accurate analysis of Jackson rule (see [11])reveals that

mJ ðmÞ6mðmÞ þ pmax;

where pmax is the maximum processing time. Atrivial lower bound on the optimal value m(m) isgiven by mðmÞP

Pmj¼1pj. Therefore,

PrfmJðmÞ=mðmÞP 1 þ 4=mg

6 PrXm

j¼1

pj � l6 � ðl � bm=4Þ( )

;

where l ¼ EPm

j¼1pjh i

¼ mb2. The claim follows by

applying Chernoff bound

PrXm

j¼1

pj � l6 � ðl � 4=mÞ" #

6 exp � 2ðl � bm=4Þ2

mb2

!6 expð�m=8Þ:


References

[1] R. Barr, B.L. Golden, J.P. Kelly, M.G.C. Resende, W.R.Stewart, Designing and reporting on computational exper-iments with heuristic methods, Journal of Heuristics 1(1995) 9–32.

[2] L. Berman, J. Hartmanis, On isomorphisms and density ofNP and other complete sets, SIAM Journal on Computing6 (2) (1977) 305–322.

[3] J. Carlier, The one-machine sequencing problem, Euro-pean Journal of Operational Research 11 (1982) 42–47.

[4] M. Cesati, L. Trevisan, On the efficiency of polynomialtime approximation schemes, Information Processing Let-ters 64 (47) (1997) 165–171.

[5] W. Clocksin, C. Mellish, Programming in Prolog,Springer-Verlag, 1981.

[6] R. Downey, M. Fellows, Parameterized Complexity,Springer, 1998.

[7] A. Fishkin, K. Jansen, M. Mastrolilli, Grouping tech-niques for scheduling problems: Simpler and faster, in: 9thAnnual European Symposium on Algorithms (ESA�01),vol. LNCS 2161, 2001, pp. 206–217.

[8] R. Graham, E. Lawler, J. Lenstra, A.R. Kan, Optimizationand Approximation in Deterministic Sequencing andScheduling: A survey, Annals of Discrete Mathematics,vol. 5, North-Holland, 1979 pp. 287–326.

[9] H. Greenberg, Computational testing: Why, how and howmuch, ORSA Journal of Computing 2 (1990) 94–97.

[10] L. Hall, D. Shmoys, Approximation algorithms for con-strained scheduling problems, in: Proceedings of the 30thIEEE Symposium on Foundations of Computer Science,1989, pp. 134–139.

[11] D. Hochbaum (Ed.), Approximation Algorithms for NP-hard Problems, PWS Publishing Company, Boston, 1995.

[12] J. Hooker, Needed: An empirical science of algorithms,Operations Research 42 (1994) 201–212.

[13] J. Hooker, Testing heuristics: We have it all wrong, Journalof Heuristics 1 (1995) 33–42.

[14] J. Jackson, Scheduling a production line to minimizemaximum tardiness, Technical Report Research Report43, Management Science Research Project, UCLA, 1955.

[15] D.S. Johnson, A theoretician�s guide to the experimentalanalysis of algorithms, in: M.H. Goldwasser, D.S. John-son, C.C. McGeoch (Eds.), Data Structures, Near Neigh-bor Searches, and Methodology: Fifth and Sixth DIMACSImplementation Challenges, American Mathematical Soci-ety, Providence, RI, 2002, pp. 215–250.

[16] J. Lenstra, A.R. Kan, P. Brucker, Complexity of MachineScheduling Problems, Annals of Operations Research 1(1997) 343–362.

[17] S. Mahaney, Sparse complete sets for NP: Solution of aconjecture of Berman and Hartmanis, Journal of Compu-ter and System Sciences 25 (2) (1982) 130–143.

[18] M. Mastrolilli, Grouping techniques for one machinescheduling subject to precedence constraints, in: Proceed-ings of the 21st Foundations of Software Technology andTheoretical Computer Science, vol. LNCS 2245, 2001,pp. 268–279.

[19] C. McGeoch, Toward an experimental method for algo-rithm simulation, INFORMS Journal of Computing 8(1996) 1–15.

[20] R. Motwani, P. Raghavan, Randomized Algorithms,Cambridge University Press, Cambridge, 1995.

[21] P. Schuurman, G. Woeginger, Approximation schemes––Atutorial, 2001.

[22] Spielman, Teng., Smoothed analysis of algorithms: Whythe simplex algorithm usually takes polynomial time, in:Proceedings of STOC 2001, pp. 296–305.

Documents

Core instances for testing: A case study