Towards Operator-less Data Centers Through Data-Driven,

Predictive, Proactive Autonomics

Alina Sırbu1 and Ozalp Babaoglu 2

1Department of Computer Science, University of Pisa, Italy2Department of Computer Science and Engineering, University of Bologna, Italy

Abstract

Continued reliance on human operators for manag-ing data centers is a major impediment for themfrom ever reaching extreme dimensions. Large com-puter systems in general, and data centers in particu-lar, will ultimately be managed using predictive com-putational and executable models obtained throughdata-science tools, and at that point, the interventionof humans will be limited to setting high-level goalsand policies rather than performing low-level opera-tions. Data-driven autonomics, where managementand control are based on holistic predictive modelsthat are built and updated using live data, opens onepossible path towards limiting the role of operators indata centers. In this paper, we present a data-sciencestudy of a public Google dataset collected in a 12K-node cluster with the goal of building and evaluatingpredictive models for node failures. Our results sup-port the practicality of a data-driven approach byshowing the effectiveness of predictive models basedon data found in typical data center logs. We use Big-Query, the big data SQL platform from the GoogleCloud suite, to process massive amounts of data andgenerate a rich feature set characterizing node stateover time. We describe how an ensemble classifiercan be built out of many Random Forest classifierseach trained on these features, to predict if nodeswill fail in a future 24-hour window. Our evaluationreveals that if we limit false positive rates to 5%,we can achieve true positive rates between 27% and88% with precision varying between 50% and 72%.This level of performance allows us to recover large

fraction of jobs’ executions (by redirecting them toother nodes when a failure of the present node is pre-dicted) that would otherwise have been wasted dueto failures. We discuss the feasibility of including ourpredictive model as the central component of a data-driven autonomic manager and operating it on-linewith live data streams (rather than off-line on datalogs). All of the scripts used for BigQuery and clas-sification analyses are publicly available on GitHub.

Keywords: Data science, predictive analytics,Google cluster trace, log data analysis, failure predic-tion, machine learning classification, ensemble classi-fier, random forest, BigQuery

1 Introduction

Modern data centers are the engines of the Inter-net that run e-commerce sites, cloud-based servicesaccessed from mobile devices and power the socialnetworks utilized each day by hundreds of millionsof users. Given the pervasiveness of these servicesin many aspects of our daily lives, continued avail-ability of data centers is critical. And when contin-ued availability is not possible, service degradationsand outages need to be foreseen in a timely man-ner so as to minimize their impact on users. For themost part, current automated data center manage-ment tools are limited to low-level infrastructure pro-visioning, resource allocation, scheduling or monitor-ing tasks with no predictive capabilities. This leavesthe brunt of the problem in detecting and resolvingundesired behaviors to armies of operators who con-

1

arX

iv:1

606.

0445

6v1

[cs

.DC

] 1

4 Ju

n 20

16

tinuously monitor streams of data being displayedon monitors. Even at the highly optimistic rate of26,000 servers managed per staffer1, this situation isnot sustainable if data centers are ever to be scaledto extreme dimensions. Applying traditional auto-nomic computing techniques to large data centers isproblematic since their complex system characteris-tics prohibit building a “cause-effect” system modelthat is essential for closing the control loop. Fur-thermore, current autonomic computing technologiesare reactive and try to steer the system back to de-sired states only after undesirable states are actuallyentered — they lack predictive capabilities to antici-pate undesirable states in advance so that proactiveactions can be taken to avoid them in the first place.

If data centers are the engines of the Internet, thendata is their fuel and exhaust. Data centers gener-ate and store vast amounts of data in the form oflogs corresponding to various events and errors in thecourse of their operation. When these computing in-frastructure logs are augmented with numerous otherinternal and external data channels including powersupply, cooling, management actions such as softwareupdates, server additions/removal, configuration pa-rameter changes, network topology modifications, oroperator actions to modify electrical wiring or changethe physical locations of racks/server/storage de-vices, data centers become ripe to benefit from datascience. The grand challenge is to exploit the toolsetof modern data science and develop a new generationof autonomics, which we call Autonomics 2.0 , that isdata-driven, predictive and proactive based on holis-tic models that capture a data centre as an ecosystemincluding not only the computer system as such, butalso its physical as well as its socio-political environ-ment.

In this paper we present the results of an initialstudy of predictive models for node failures in datacenters. Such models will be an integral part of an“Autonomics 2.0” architecture that will also includepredictive models built from other data sources aswell as components to enact proactive control. Ourstudy is based on a recent Google dataset containing

1Delfina Eberly, Director of Data Center Operations atFacebook, speaking on “Operations at Scale” at the 7x24 Ex-change 2013 Fall Conference.

workload and scheduler events emitted by the Borgcluster management system [34] in a cluster of over12,000 nodes during a one-month period [36, 23]. Weemployed BigQuery [33], a big data tool from theGoogle Cloud Platform that allows SQL-like queriesto be run on massive data, to perform an exploratoryfeature analysis. This step generated a large numberof features at various levels of aggregation suitablefor use in a machine learning classifier. The use ofBigQuery has allowed us to complete the analysis forlarge amounts of data (table sizes up to 12TB con-taining over 100 billion rows) in reasonable amountsof time.

For the classification study, we employed an en-semble that combines the output of multiple RandomForests (RF) classifiers, which themselves are ensem-bles of Decision Trees. RF were employed due totheir proven suitability in situations were the num-ber of features is large [24] and the classes are “un-balanced” [15] such that one of the classes consistsmainly of “rare events” that occur with very low fre-quency. Although individual RF were better thanother classifiers that were considered in our initialtests, they still exhibited limited performance, whichprompted us to pursue an ensemble approach. Whileindividual trees in RF are based on subsets of fea-tures, we used a combination of bagging and datasubsampling to build the RF ensemble and tailor themethodology to this particular dataset. Our ensem-ble classifier was tested on several days from the tracedata, resulting in very good performance on somedays (up to 88% true positive rate, TPR, and 5%false positive rate, FPR), and modest performanceon other days (minimum of 27% TPR at the same 5%FPR). Precision levels in all cases remained between50% and 72%. We should note that these results arecomparable to other failure prediction studies in thefield.

The contributions of our work are severalfold.First, we argue that modern data centers can bescaled to extreme dimensions only after eliminatingreliance on human operators by adopting a new gen-eration of autonomics that is data-driven and basedon holistic predictive models. Towards this goal,we present a failure prediction analysis based on adataset that has been studied extensively in the lit-

2

erature from other perspectives. Next, we propose anensemble classification methodology tailored to thisparticular problem where subsampling is combinedwith bagging and precision-weighted voting to maxi-mize performance. Finally, we provide one of the firstinstances of BigQuery usage in the literature withquantitative evaluation of running times as a func-tion of data size. Our results show that models withsufficient predictive powers can be built based on datafound in typical logs and that they can form the basisof an effective data-driven, predictive and proactiveautonomic manager. All of the scripts used for Big-Query and classification analysis are publicly avail-able on GitHub [32] under the GNU General PublicLicense.

The rest of this paper is organized as follows. Thenext section describes the process of building featuresfrom the trace data. Section 3 describes our classifi-cation approach while our prediction results are pre-sented in Section 4. Impact on wasted resources is es-timated in Section 5 while in Section 6 we discuss theissues surrounding the construction of a data-drivenautonomic controller based on our predictive modeland argue its practicality. Related work is discussedin Section 7 while Section 8 concludes the paper.

2 Building the feature set withBigQuery

The workload trace published by Google contains sev-eral tables monitoring the status of nodes, jobs andtasks during a period of approximately 29 days for acluster of 12,453 nodes. This includes task events(over 100 million records, 17GB uncompressed),which follow the state evolution for each task, andtask usage logs (over 1 billion records, 178GB un-compressed), which report the amount of resourcesper task at approximately 5 minute intervals. Wehave used the data to compute the overall load andstatus of different cluster nodes at 5 minute intervals.This resulted in a time series for each node and fea-ture that spans the entire trace (periods when thenode was “up”). We then proceeded to obtain sev-eral features by aggregating measures in the original

data. Due to the size of the dataset, this aggrega-tion analysis was performed using BigQuery on thetrace data directly from Google Cloud Storage. Weused the bq command line tool for the entire analysis,and our scripts are available online through our Website [32].

From task events, we obtained several time seriesfor each node with a time resolution of 5 minutes.A total of 7 features were extracted, which countthe number of tasks currently running, the numberof tasks that have started in the last 5 minutes andthose that have finished with different exit statuses —evicted, failed, finished normally, killed or lost. Fromtask usage data, we obtained 5 additional features(again at 5-minute intervals) measuring the load atnode level in terms of: CPU, memory, disk time, cy-cles per instruction (CPI) and memory accesses perinstruction (MAI). This resulted in a total of 12 ba-sic features that were extracted. For each feature, ateach time step we consider the previous 6 time win-dows (corresponding to the node status during thelast 30 minutes) obtaining 72 features in total (12basic features × 6 time windows).

The procedure for obtaining the basic features wasextremely fast on the BigQuery platform. For taskcounts, we started with constructing a table of run-ning tasks, where each row corresponds to one taskand includes its start time, end time, end status andthe bide it was running on. Starting from this table,we could obtain the time series for each feature foreach node, requiring between 139 and 939 secondson BigQuery per feature (one separate table per fea-ture was obtained). The features related to node loadwere computed by summing over all tasks running ona node in each time window, requiring between 3585and 9096 seconds on BigQuery per feature. The in-creased execution time is due to the increased tablesizes (over 1 billion rows). We then performed a JOINof all above tables to combine the basic features into asingle table with 104,197,215 rows (occupying 7GB).For this analysis, our experience allows us to judgeBigQuery as being extremely fast; an equivalent com-putation would have taken months to perform on aregular PC.

A second level of aggregation meant looking at fea-tures over longer time windows rather than just the

3

Aggregation Average, SD, CV Correlation1h 166 (all features) 45(6.5)12h 864 (all features) 258.8(89.1)24h 284.6(86.6) 395.6(78.9)48h 593.6(399.2) 987.2(590)72h 726.6(411.5) 1055.47(265.23)96h 739.4(319.4) 1489.2(805.9)

Table 1: Running times required by BigQuery for ob-taining features aggregated over different time win-dows, for two aggregation types: computing averages,standard deviation (SD) and coefficient of variation(CV) versus computing correlations. For 1h and 12hwindows, average, SD and CV were computed for allfeatures in a single query. For all other cases, themean (and standard deviation) of the required timesper feature are shown.

last 5 minutes. At each time step, 3 different statis-tics — averages, standard deviations and coefficientsof variation — were computed for each basic featureobtained at the previous step. This was motivatedby the suspicion that not only feature values but alsotheir deviation from the mean could be important inunderstanding system behavior. Six different runningwindows of sizes 1, 12, 24, 48, 72 and 96 hours wereused to capture behavior at various time resolutions.This resulted in 216 additional features (3 statistics× 12 features × 6 window sizes).

In order to generate these aggregated features, aset of intermediate tables were used. For each timepoint, these tables consisted of the entire set of datapoints to be averaged. For instance, for 1-hour aver-ages, the table would contain a set of 6 values for eachfeature and for each time point, showing the evolu-tion of the system over the past hour. While gener-ating these tables was not time consuming (requiringbetween 197 and 960 seconds), their sizes were quiteimpressive: ranging from 143 GB (over 1 billion rows)for 1 hour up to 12.5 TB (over 100 billion rows) inthe case of 96-hour window. Processing these tablesto obtain the aggregated features of interest requiredsignificant resources and would not have been pos-sible without BigQuery. Even then, direct queriesusing a single group by operation to obtain all 216

features was not possible, requiring only one basicfeature to be handled at a time and combining theresults into a single table at the end. Table 1 showsstatistics over the time required to obtain one featurefor the different window sizes.

Although independent feature values are impor-tant, another criterion that could be important forprediction is the relations that exist between differentmeasures. Correlation between features is one suchmeasure, with different correlation values indicatingchanges in system behavior. Hence we introduceda third level of aggregation of the data by comput-ing correlations between a chosen set of feature pairs,again over various window sizes (1 to 96 hours as be-fore). We chose 7 features to analyze: number ofrunning, started and failed jobs together with CPU,memory, disk time and CPI. By computing correla-tions between all possible pairings of the 7 features,we obtained a total of 21 correlation values for eachwindow size. This introduces 126 additional featuresto our dataset. The BigQuery analysis started fromthe same intermediate tables as before and computedcorrelations for one feature pair at a time. As can beseen in Table 1, this step was more time consum-ing, requiring greater time than the previous aggre-gation step, yet still remains manageable consideringthe size of the data. The amount of data processedfor these queries ranged from 49.6GB (per featurepair for 1-hour windows) to 4.33TB (per feature pairfor 96-hour windows), resulting in a higher processingcost (5 USD per TB processed). Yet again, a simi-lar analysis would not have been possible without theBigQuery platform.

The Google trace also reports node events. Theseare scheduler events corresponding to nodes beingadded or removed from the pool of resources. Ofparticular interest are remove events, which can bedue to two causes: node failures or software updates.The goal of this work is to predict remove events dueto node failures, so the two causes have to be distin-guished. Prompted by our discussions, publishers ofthe Google trace investigated the best way to performthis distinction and suggested to look at the lengthof time that nodes remain down — the time from theremove event of interest to the next add event forthe same node. If this “down time” is large, then we

4

can assume that the remove event was due to a nodefailure, while if it is small, the node was most likelyremoved for a software update. To ensure that anevent considered to be a failure is indeed a real fail-ure, we used a relatively-long “down time” thresholdof 2 hours, which is greater than the time required fora typical software update. Based on this threshold,out of a total of 8,957 remove events, 2,298 were con-sidered failures, and were the target of our predictivestudy. For the rest of the events, for which we cannotbe sure of the cause, the data points in the preceding24-hour window were removed completely from thedataset. An alternative would have been consideringthem part of the safe class, however this might notbe true for some of the points. Thus, removing themcompletely ensures that all data labeled as safe arein fact safe.

To the above features based mostly on load mea-sures, we added two new features: the up time foreach node (time since the last corresponding addevent) and number of remove events for the entirecluster within the last hour. This resulted in a totalof 416 features for 104,197,215 data points (almost300GB of processed data).

3 Classification approach

The features obtained in the previous section wereused for classification with the Random Forest (RF)classifier. The data points were separated into twoclasses: safe (negatives) and fail (positives). Todo this, for each data point (corresponding to onenode at a given time) we computed time to removeas the time to the next remove event. Then, allpoints with time to remove less than 24 hours wereassigned to the class fail while all others were as-signed to the class safe. We extracted all the faildata points corresponding to real failures (108,365data points) together with a subset of the safe class,corresponding to 0.5% of the total by random sub-sampling (544,985 points after subsampling). Weused this procedure to deal with the fact that thesafe class is much larger than the fail class andclassifiers have difficulty learning patterns from veryimbalanced datasets. Subsampling is one way of re-

ducing the extent of this imbalance [11]. Even afterthis subsampling procedure, negatives are about fivetimes the number of positives. These 653,350 datapoints (safe plus fail) formed the basis of our pre-dictive study.

Given the large number of features, some mightbe more useful than others, hence we explored twotypes of feature selection mechanisms. One was prin-cipal component analysis (PCA), using the originalfeatures to build a set of principal components —additional features that account for most of the vari-ability in the data. Then one can use only the topprincipal components for classification, since thoseshould contain the most important information. Wetrained classifiers with an increasing number of prin-cipal components, however the performance obtainedwas not better than using the original features. Asecond mechanism was to filter the original featuresbased on their correlation to the time to the nextfailure event (time to remove above). Correlationswere in the interval [−0.3, 0.45], and we used onlythose features with absolute correlation larger than athreshold. We found that the best performance wasobtained with a null threshold, which means againusing all features. Hence, our attempts to reduce thefeature set did not produce better results that the RFtrained directly on the original features. One reasonfor this may be the fact that the RF itself performsfeature selection when training the Decision Trees. Itappears that the RF mechanism performs better inthis case than correlation-based filtering or PCA.

To evaluate the performance of our approach, weemployed cross validation. Given the procedure weused to define the two classes, there are multiple datapoints corresponding to the same failure (data over24 hours with 5 minutes resolution). Since some ofthese data points are very similar, choosing the trainand test data cannot be done by selecting randomsubsets. While random selection may give extremelygood prediction results, it is not realistic since wewould be using test data which is too similar to thetraining data. This is why we opted for a time-basedseparation of train and test data. We considered bas-ing the training on data over a 10-day window, fol-lowed by testing based on data over the next daywith no overlap with the training data. Hence, the

5

Benchmark 1

Training data Test data

Benchmark 2

Benchmark 15…

Trace days

Figure 1: Cross validation approach: forward-in-timetesting. Ten days were used for training and one dayfor testing. A set of 15 benchmarks (train/test pairs)were obtained by sliding the train/test window overthe 29-day trace.

test day started 24 hours after the last training datapoint. The first two days were omitted in order to de-crease the effect on aggregated features. In this man-ner, fifteen train/test pairs were obtained and usedas benchmarks to evaluate our analysis (see Fig. 1).This forward-in-time cross validation procedure en-sures that classification performance is realistic andnot an artifact of the structure of the data. Also, itmimics the way failure prediction would be appliedin a live data center, where every day a model couldbe trained on past data to predict future failures.

Given that many points from the fail class arevery similar, which is not the case for the safeclass due to initial subsampling, the information inthe safe class is still overwhelmingly large. Thisprompted us to further subsample the negative classin order to obtain the training data. This wasperformed in such a way that the ratio betweensafe and fail data points is equal to a parame-ter fsafe. We varied this parameter with the values{0.25, 0.5, 1, 2, 3, 4} while using all of the data pointsfrom the positive class so as not to miss any usefulinformation. This applied only for training data: fortesting we always used all data from both the nega-tive and positive classes (out of the base dataset of653,350 points). We also used RF of different sizes,with the number of Decision Trees varying from 2 to15 with a step of 1 (resulting in 14 different values).

As we will discuss in the next section, the perfor-mance of the individual classifiers, while better thanrandom, was judged to be not satisfactory. Hencewe opted for an ensemble method, which builds a se-

ries of classifiers and then selects and combines themto provide the final classification. Ensembles can en-hance the power of low performing individual clas-sifiers [24], especially if these are diverse [16, 30]: ifthey give false answers on different data points (in-dependent errors), then combining their knowledgecan improve accuracy. To create diverse classifiers,one can vary model parameters but also train themwith different data (known as the bagging method[24]). Bagging matches very well with subsamplingto overcome the rare events issue, and it has beenshown to be effective for the class-imbalance prob-lem [11]. Hence, we adopt a similar approach to buildour individual classifiers. Every time a new classifieris trained, a new training dataset is built by consid-ering all the data points in the positive class and arandom subset of the negative class. As describedearlier, the size of this subset is defined by the fsafeparameter. By varying the value of fsafe and numberof trees in the RF algorithm, we created diverse clas-sifiers. The following algorithm details the procedureof building the individual classifiers in the ensemble.

Require: train pos, train neg, Shuffle(), Train()fsafe ← {0.25, 0.5, 1, 2, 3, 4}tree count ← {2..15}classifiers ← {}start ← 0for all fs ∈ fsafe do

for all tc ∈ tree count doend ← start + fs ∗ |train pos|if end ≥ |train neg | then

start ← 0end ← start + fs ∗ |train pos|Shuffle(train neg)

end iftrain data ← train pos + train neg [start : end ]classifier ← Train(train data, tc)append(classifiers, classifier)start ← end

end forend for

We repeated this procedure 5 times, resulting in5 classifiers for each combination of the parametersfsafe and RF size. This resulted in a total of 420 RFin the ensemble (5 repetitions × 6 fsafe values × 14

6

RF sizes).

Once the pool of classifiers is obtained, a combin-ing strategy has to be used. Most existing approachesuse the majority vote rule — each classifier votes onthe class and the majority class becomes the finaldecision [24]. Alternatively, a weighted vote can beused, and we opted for precision-weighted voting. Formost existing methods, weights correspond to the ac-curacy of each classifier on training data [20]. In ourcase, performance on training data is close to perfectand accuracy is generally high, which is why we useprecision on a subset of the test data. Specifically,we divide the test data into two halves: an individ-ual test data set and an ensemble test data set. Theformer is used to evaluate the precision of individualclassifiers and obtain a weight for their vote. Thelatter provides the final evaluation of the ensemble.All data corresponding to the test day was used, withno subsampling. Table 2 shows the number of datapoints used for each benchmark for training and test-ing. While the parameter fsafe controlled the ratiosafe/fail during training, fail instances were muchless frequent during testing, varying between 13% and36% of the number of safe instances.

To perform precision-weighted voting, we first ap-plied each RF i obtained above to the individual testdata and computed their precision pi as the fractionof points labeled fail that were actually failures. Inother words, precision is the probability that an in-stance labeled as a failure is actually a real failure,which is why we decided to use this as a weight. Thenwe applied each RF to the ensemble test data. Foreach data point j in this set, each RF provided a clas-sification oji (either 0 or 1 corresponding to safe orfail, respectively). The classification of the ensem-ble (the whole set of RF) was then computed as acontinuous score

sj =∑i

ojipi (1)

by summing individual answers weighted by theirprecision. Finally, these were normalized by the high-est score in the data

s′j =sj

maxj(sj)(2)

Train Individual Test Ensemble TestBenchmark fail fail safe fail safe

1 41485 2055 9609 2055 96102 41005 2010 9408 2011 94083 41592 1638 9606 1638 96064 42347 1770 9597 1770 95985 42958 1909 9589 1909 95896 42862 1999 9913 2000 99147 41984 1787 9821 1787 98228 39953 1520 10424 1520 104249 37719 1665 10007 1666 1000810 36818 1582 9462 1583 946311 35431 1999 9302 1999 930212 35978 3786 10409 3787 1041013 35862 2114 9575 2114 957514 39426 1449 9609 1450 961015 40377 1284 9783 1285 9784

Table 2: Size of training and testing datasets. Fortraining data, the number of safe data points is thenumber of fail multiplied by the fsafe parameter ateach run.

The resulting score s′j is proportional to the likeli-hood that a data point is in the fail class — thehigher the score, the more certain we are that wehave an actual failure. The following algorithm out-lines the procedure of obtaining the final ensembleclassification scores.

Require: classifiers, individual test, ensemble testRequire: Precision(), Classify()

classification scores ← {}weights ← {}for all c ∈ classifiers do

w ← Precision(c, individual test)weights[c] ← w

end forfor all d ∈ ensemble test do

score ← 0for all c ∈ classifiers do

score ← score+weights[c]∗Classify(c, d)end forappend(classification scores, score)

end formax ← Max (classification scores)

7

for all s ∈ classification scores dos ← s/max

end for

It assumes that the set of classifiers is available(classifiers), together with the two test data sets (in-dividual test and ensemble test) and procedures tocompute precision of a classifier on a dataset (Pre-cision()) and to apply a classifier to a data point(Classify() which returns 0 for safe and 1 for fail).

4 Classification results

The ensemble classifier was applied to all 15 bench-mark datasets. Training was done on an iMac with3.06GHz Intel Core 2 Duo processor and 8GB of1067MHz DDR3 memory running OSX 10.9.3. Train-ing of the entire ensemble took between 7 and 9 hoursfor each benchmark.

Given that the result of the classification is a con-tinuous score (Equation 2), and not a discrete label,evaluation was based on the Receiver Operating Char-acteristic (ROC) and Precision-Recall (PR) curves.A class can be obtained for a data point j from thescore s′j by using a threshold s∗. A data point is con-sidered to be in the fail class if s′j ≥ s∗. The smallers∗, the more instances are classified as failures. Thus,by decreasing s∗ the number of true positives growsbut so do the false positives. Similarly, at differentthreshold values, a certain precision is obtained. TheROC curve plots the True Positive Rate (TPR) ver-

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Benchmark

0.0

0.2

0.4

0.6

0.8

1.0 AUROCAUPR

Figure 2: AUROC and AUPR on ensemble test data.

sus the False Positive Rate (FPR) of the classifieras the threshold is varied. Similarly, The PR curvedisplays the precision versus recall (equal to TPR orSensitivity). It is common to evaluate a classifier bycomputing the area under ROC (AUROC) and areaunder PR (AUPR) curves, which can range from 0to 1. AUROC values greater than 0.5 correspond toclassifiers that perform better than random guesses,while AUPR represents an average classification pre-cision, so, again, the higher the better. AUROC andAUPR do not depend on the relative distribution ofthe two classes, so they are particularly suitable forclass-imbalance problems such as the one at hand.

Fig. 2 shows AUROC and AUPR values obtainedfor all datasets, evaluated on the ensemble test data.For all benchmarks, AUROC values are very good,over 0.75 and up to 0.97. AUPR ranges between 0.38and 0.87. Performance appears to increase, especiallyin terms of precision, towards the end of the trace.Lower performance that is observed for the first twobenchmarks could be due to the fact that some ofthe aggregated features (those over 3 or 4 days) arecomputed with incomplete data at the beginning.

To evaluate the effect of the different parametersand the ensemble approach, Fig. 3 displays the ROCand PR curves for the benchmarks that result in theworst and best results (4 and 14, respectively). Per-formance of the individual classifiers in the ensembleare also displayed (as points in the ROC/PR spacesince their answer is categorical). We can see thatindividual classifiers result in very low FPR which isvery important in predicting failures. Yet, in manycases, the TPR values are also very low. This meansthat most test data is classified as safe and veryfew failures are actually identified. TPR appears toincrease when the fsafe parameter decreases, but atthe expense of the FPR and Precision. The plotsshow quantitatively the clear dependence betweenthe three plotted measures and fsafe values. As theamount of safe training data decreases, the clas-sifiers become less stringent and can identify morefailures, which is an important result for this class-imbalance problem. Also, the plot shows clearly thatindividual classifiers obtained with different values forfsafe are diverse, which is critical for obtaining goodensemble performance.

8

(a) Worst case (Benchmark 4)

(b) Best case (Benchmark 14)

Figure 3: ROC and PR curves for worst and best performance across the 15 benchmarks (4 and 14, respec-tively). The vertical lines correspond to FPR of 1%, 5% and 10%. Note that parameter fsafe controls theratio of safe to fail data in the training datasets.

In general, the points corresponding to the indi-vidual classifiers are below the ROC and PR curvesdescribing the performance of the ensemble. Thisproves that the ensemble method is better than theindividual classifiers for this problem, which can bealso due to their diversity. Some exceptions do ap-pear (points above the solid lines), however for verylow TPR (under 0.2) so in an area of the ROC/PR

space that is not interesting from our point of view.We are interested in maximizing the TPR while keep-ing the FPR at bay. At a FPR of 5%, which meansfew false alarms, the two examples from Fig. 3 dis-play TPR values of 0.272 (worst case) and 0.886 (bestcase), corresponding to precision values of 0.502 and0.728 respectively. This is much better than individ-ual classifiers at this level, both in terms of precision

9

and TPR. For failure prediction, this means that be-tween 27.2% and 88.6% of failures are identified assuch, while from all instances labeled as failures, be-tween 50.2% and 72.8% are actual failures.

According to our classification strategy, a nodewould be considered to be in safe state whether itfails in 2 days or in 2 weeks. Similarly, it is con-sidered to be in fail state whether it fails in 10 min-utes or within 23 hours. Obviously the two situationsare very different and the impact of misclassificationvaries depending on the time to the next failure. Inearlier work [31], we have shown that misclassifica-tions of time points in the fail class (false negatives)are in general far from the time of failure, while mis-classifications of the safe class (false positives or falsealarms) are, on average, closer to the failure instantthan are true negatives (correct classification of safetime points). This implies that the impact of misclas-sification is reduced.

5 Impact on running tasks

When a node fails, all tasks running on that node areinterrupted. Thus, resources (e.g., CPU time) thathad been consumed by interrupted tasks are wasted.To study the extent of this wastage, we estimated thenumber of interrupted tasks and the correspondingCPU-hours wasted due to failures in our data. Thestudy was limited to the period of the trace corre-sponding to ensemble test data in all 15 benchmarksrepresenting 180 hours of data and containing 668node failures. For this period, a simple count of tasksevicted or killed in close vicinity of the failure resultsin a total of 5,488 interrupted tasks, correspondingto over 31,393 CPU-hours being wasted.

The Google dataset contains data from differ-ent tasks that use resources very differently. Sometasks correspond to latency-sensitive, long-running,revenue-generating production services that respondto user queries. If these tasks are interrupted, onlythe latest queries will be affected, so most of theCPU time they used is not actually wasted. Othertasks are from non-production batch jobs, and typi-cally return a value at the very end, so their interrup-tion results in all of the CPU time they used being

wasted. The Google dataset includes the schedulingclass of a task to distinguish between production ser-vices (higher scheduling classes) and non-productionbatch jobs (lower scheduling classes). Table 3 indi-cates the distribution of the interrupted tasks into thedifferent scheduling classes. We can see that mostinterrupted tasks are from scheduling class 0 (non-production). However, most of the wasted CPU timecorresponds to higher scheduling classes since thesejobs are long running.

Class Interrupted Recovered Redirected0 2,260(2,675) 1,863(1,538) 27,908(6,550)1 851(1,748) 387(351) 4,796(2,294)2 1,632(16,663) 667(2,035) 7,586(5,615)3 745(10,305) 185(1,269) 791(3584)

Total 5,488(31,393) 3,102(5,194) 41,081(18,044)

Table 3: Number of tasks in each scheduling classthat are interrupted by node failures, together withthose recovered and redirected with perfect predic-tion. CPU-hours used by the tasks in each categoryare shown in parentheses (wasted, recovered and redi-rected CPU-hours).

Predicting failures in advance can help reduce re-source wastage by modifying the resource allocationdecisions dynamically. This is in itself an extensiveresearch area, but a simple procedure could be toquarantine nodes that are predicted to fail by notsubmitting any new jobs to them for some period oftime. If consecutive failure alarms continue to ap-pear, the quarantine period is extended until eitherthe alarms stop or the node fails. In the following, wesimulate such an approach. For better precision, wequarantine a node only if two consecutive time pointsare classified as FAIL.

While a node is in quarantine, all tasks that wouldhave otherwise run on that node need to be redirected.Among redirected tasks, some would have finishedbefore the node failure, others would have been in-terrupted. We call the latter recovered tasks, sincetheir interruption was avoided by proactively redi-recting them. The aim of our proactive approach isto maximize the number of recovered tasks (the gain)while minimizing the number of redirected tasks (the

10

(a) Tasks recovered (b) CPU time recovered

(c) Tasks redirected (d) CPU time redirected

Figure 4: Number of tasks and CPU hours redirected and recovered after prediction, compared to thebaseline given by perfect prediction results.

11

cost). As we shall see later in this section, the twoobjectives are contradictory: the number of recov-ered tasks grows as the number of redirected tasksincreases, so there is a tradeoff between the cost andthe gain.

To provide a baseline for our results, we estimatedthe number of recovered tasks if perfect predictionwere possible 24 hours in advance on our data. Per-fect prediction means that a failure alarm is trig-gered every 5 minutes, starting 24 hours before thefailure. Hence, nodes will enter quarantine state 24hours before their failure and remain in quarantineuntil they fail. Since lead time for failure predictionis limited (for us it is 24 hours), not all interruptedtasks can be recovered (those that started before thefirst failure alarm). Table 3 shows the number of in-terrupted, recovered and redirected tasks along withthe respective CPU-hours, globally and according toscheduling class, assuming perfect prediction. Sincetasks in lower scheduling classes are shorter, they areeasier to recover since they can start after the fail-ure alarms. In particular, 82.4% of the interruptedtasks of scheduling class 0 and only 24.8% of schedul-ing class 3 can be recovered by perfect prediction,corresponding to recovering 57.5% and 12.3% CPU-hours, respectively. As discussed earlier, for higherscheduling classes, task interruption does not neces-sarily mean that all used resources are wasted. Thismakes estimating the exact resource wastage for highscheduling class tasks very difficult. Hence, in thefollowing we limit our analysis to tasks from schedul-ing class 0, where all resources used by interruptedtasks can be considered wasted.

Taking Table 3 (i.e., perfect prediction) as a base-line, we evaluated the effect of our prediction methodon running tasks. When using our classifier model forpredictions, we may have false alarms resulting in anode to be quarantined without reason, while otheralarms may be missed, so the node may not be quar-antined for an entire 24 hours before it fails. In orderto study the tradeoff between cost and gain, we ex-plored various quarantine windows and different FalsePositive Rates (FPR) that characterize our classifica-tion. Ideally, our method should be able to recovera large percentage of the baseline recovered tasks,while not exceeding the baseline redirected tasks sig-

nificantly.

Fig. 4 shows the number of recovered and redirectedtasks for our method when limited to scheduling class0, along with the corresponding CPU-hours. Thenumber of recovered tasks increases with increasingthe FPR and the quarantine windows, however at thecost of increasing the total number of redirected tasks.A quarantine window size of 4 hours and FPR = 0.2appears to provide a good compromise between recov-ered and redirected tasks, with about 48% of baselinetasks and 50% of baseline CPU time recovered and102% of baseline tasks and 96% baseline CPU hoursredirected. Hence our technique can have a significantimpact on running tasks with respect to the baseline.A more sophisticated strategy can further enhanceboth the baseline and our classification results.

6 Discussion

We now consider how our predictive model can be thebasis for a data-driven “Autonomics 2.0” controller tobe deployed in data centers. In such a scenario, bothmodel building and model updating would have tohappen on-line on data that is being streamed fromvarious sources. We outline some of the changes re-quired by our model workflow to allow on-line use.

To compute on-line the features necessary formodel building, log data can be collected in a Big-Query table using the streaming API. As data isbeing streamed, features have to be computed at 5minute intervals. Both basic and aggregated features(averages, standard deviations, coefficients of varia-tion and correlations) have to be computed, but onlyfor the last time window (previous time windows arealready stored in a dedicated table). Basic featuresare straightforward to compute requiring negligiblerunning time since they can be computed using accu-mulators as the events come in. Aggregated featurescan be computed in parallel since they are indepen-dent of each other. In our experiments, correlationcomputation was most time consuming, with an av-erage time to compute one correlation feature overthe longest time window taking 1489.2 seconds forall values over the 29 days (Table 1). For comput-ing a single value (for the newly streamed data), the

12

time required should be on average under 0.2 sec-onds. This estimate is based on a linear dependencebetween the number of time windows and compu-tation time and offers an upper bound for the timerequired. If this stage is performed in parallel onBigQuery, this value would also be the average timeto compute all 126 correlation features (each featurecan be computed independently so speedup would belinear).

In terms of dollar costs, we expect figures simi-lar to those during our tests — about 70 USD perday for storage and analysis. To this, the stream-ing costs would have to be added — currently 1 centper 200MB. For our system, the original raw data isabout 200GB for all 29 days, so this would translateto approximately 7GB of data streamed every day fora system of similar size, resulting in about 35 cents ofadditional cost per day. At all times, only the last 12days of features need to be stored, which keeps datasize relatively low. In our analysis, for all 29 days,the final feature table requires 295GB of BigQuerystorage, so 12 days would amount to about 122GB ofdata.

When a new model has to be trained (e.g., oncea day), all necessary features are already computed.One can use an infrastructure like the Google Com-pute Engine to train the model, which would elimi-nate the need to download the data and would allowfor training of the individual classifiers of the ensem-ble in parallel. In our tests, the entire ensemble tookunder 9 hours to train, with each RF requiring atmost 3 minutes. Again, since each classifier is inde-pendent, training all classifiers in parallel would takeunder 3 minutes as well (provided one can use asmany CPUs as there are RFs — 420 in our study).Combining the classifiers requires a negligible amountof time.

All in all, we expect the entire process of updat-ing the model to take under 5 minutes if full paral-lelization is used both for feature computation andtraining. Application of the model on new data re-quires a negligible amount of time once features areavailable. This makes the method very practical foron-line use. Here we have described a cloud com-puting scenario, however, given the relatively limitedcomputation and storage resources that are required,

we believe that more modest clusters can also be usedfor monitoring, model updating and prediction.

7 Related work

The publication of the Google trace data has trig-gered a flurry of activity within the community in-cluding several with goals that are related to ours.Some of these provide general characterization andstatistics about the workload and node state for thecluster [21, 22, 18] and identify high levels of het-erogeneity and dynamism in the system, especiallywhen compared to grid workloads [6]. User profiles[1] and task usage shapes [37] have also been char-acterized for this cluster. Other studies have appliedclustering techniques for workload characterization,either in terms of jobs and resources [19, 35] or place-ment constraints [29], with the aim to synthesize newtraces. Wasted resources due to the priority-basedeviction mechanism were evaluated in [25], were sig-nificant resources were shown to be used by tasks thatdo not complete successfully, as we have also seen inour analysis of tasks interrupted by failures.

A different class of studies address validation ofvarious workload management algorithms. Examplesinclude [13] where the trace is used to evaluate consol-idation strategies, [5, 4] where over-committing (over-booking) is validated, [38] that takes heterogeneityinto account to perform provisioning or [8] investi-gating checkpointing algorithms.

System modeling and prediction studies using theGoogle trace data are far fewer than those aimedat characterization or validation. An early attemptat system modeling based on this trace validates anevent-based simulator using workload parameters ex-tracted from the data, with good performance in sim-ulating job status and overall system load [2, 3]. Hostload prediction using a Bayesian classifier was ana-lyzed in [7]. Using CPU and RAM history, the meanload in a future time window is predicted by dividingpossible load levels into 50 discrete states. Job fail-ure prediction and mitigation is attempted in [26],with 27% of wasted computational time saved bytheir method. Here we investigate predicting nodefailures for this cluster, which to our knowledge has

13

not been attempted before.Failure prediction in general has been an active re-

search area, with a comprehensive review [27] sum-marizing several methods of failure prediction in sin-gle machines, clusters, application servers, file sys-tems, hard drives, email servers and clients, etc., di-viding them into failure tracking, symptom monitor-ing or error reporting. The method introduced herefalls into the symptom monitoring category, howeverelements of failure tracking and error reporting arealso present through features like number of recentfailures and job failure events.

More recent studies concentrate on larger scale dis-tributed systems such as HPC or clouds. For failuretracking methods, an important resource is the failuretrace archive [14], a repository for failure logs and anassociated toolkit that enables integrated characteri-zation of failures, such as distributions of inter-eventtimes. Job failure in a cloud setting has been ana-lyzed in [28]. The naive Bayes classifier is used toobtain a probability of failure based on the job typeand host name. This is applied to traces from Ama-zon EC2 running several scientific applications. Themethod reaches different performances on jobs fromthree different application settings, with FNs of 4%,12% and 16% of total data points and correspondingFPs of 0%, 4% and 10% of total data points. Thiscorresponds approximately to FPR of 0%, 5.7% and16.3%, and TPR of 86.6%, 61.2% and 58.9%. Theperformance we obtained with our method is withina similar range for most benchmarks, although wenever reach their best performance. However, we arepredicting node failures rather than job failures. Arecent study of the Blue Waters HPC installation atArgonne National Laboratory predicted failures with60% TPR and 85% precision [10], again representingperformance similar to ours.

A comparison of different classification tools forfailure prediction in an IBM Blue Gene/L cluster isgiven in [17]. In this work, Reliability, Availabilityand Serviceability (RAS) events are analyzed usingSVMs, neural networks, rule based classifiers and acustom nearest neighbor algorithm, in an effort topredict whether different event categories will ap-pear. The custom nearest neighbor algorithm out-performs the others reaching 50% precision and 80%

TPR. A similar analysis was also performed for aBlue Gene/Q cluster [9]. The best performance wasagain achieved by the nearest neighbor classifier (10%FPR, 20% TPR). They never evaluated the RandomForest or ensemble algorithms.

In [12] an anomaly detection algorithm for cloudcomputing is introduced. It employs Principal Com-ponent Analysis and selects the most relevant prin-cipal components for each failure type. They findthat higher order components exhibit higher corre-lation with errors. Using a threshold on these prin-cipal components, they identify data points outsidethe normal range. They study four types of fail-ures: CPU-related, memory-related, disk-related andnetwork-related faults, in a controlled in-house sys-tem with fault injection and obtain very high perfor-mance, 91.4% TPR at 3.7% FPR. On a productiontrace (the same Google trace we are using) they pre-dict task failures at 81.5% TPR and 27% FPR (at 5%FPR, TPR is down to about 40%). In our case, westudied the same trace but looking at node failuresas opposed to task failures, and obtained TRP valuesbetween 27% and 88% at 5% FPR.

All above-mentioned failure prediction studies con-centrate on types of failures or systems different fromours and obtain variable results. In all cases, ourpredictions compare well with prior studies, with ourbest result being better than most.

8 Conclusions

We have presented study of node failure models fordata centers based on log data from a Google cluster.Such models will be an integral part of a new gener-ation of “Autonomics 2.0” architectures. Our resultsconfirm that models with sufficient predictive powerscan indeed be built based on data found in typicallogs and that they can form the basis of an effectiveautonomic manager that is data driven, predictiveand proactive.

Model feature extraction from the raw data wasperformed using BigQuery, the big data cloud plat-form from Google that supports SQL-like queries. Alarge number of features were generated and an en-semble classifier was trained on log data for 10 days

14

and tested on the following non-overlapping day. Thelength of the trace allowed repeating this process 15times producing 15 benchmark datasets, with the lastday in each dataset being used for testing.

The BigQuery platform was extremely useful forobtaining the features from log data. Although lim-its were found for join and group by statements,these were circumvented by creating intermediate ta-bles, which at times contained more than 12TB ofdata. Even so, features were obtained in a reason-able amount of time with overall cost for the entireanalysis processing a one-month log coming in at un-der 2,000 USD2, resulting in a daily cost under 70USD.

Classification performance varied from one bench-mark to another, with Area-Under-the-ROC curvemeasure varying between 0.76 and 0.97 while Area-Under-the-Precision-Recall curve measure varyingbetween 0.38 and 0.87. This corresponded to truepositive rates in the range 27%-88% and precisionvalues between 50% and 72% at a false positive rateof 5%. In other words, this means that in the worstcase, we were able to identify 27% of failures, whileif a data point was classified as a failure, we couldhave 50% confidence that we were looking at a realfailure. For the best case, we were able to identify al-most 90% of failures and 72% of instances classifiedas failures corresponded to real failures. All this, atthe cost of having a 5% false alarm rate.

Although not perfect, our predictions achieve goodperformance levels. Results could be improved bychanging the subsampling procedure. Here, only asubset of the safe data was used due to the largenumber of data points in this class, and a randomsample was extracted from this subset when trainingeach classifier in the ensemble when in fact one couldsubsample every time from the full set. However,this would require greater computational resourcesfor training, since a single workstation cannot pro-cess 300 GB of data at a time. Training times couldbe reduced through parallelization, since the prob-lem is embarrassingly parallel (each classifier in theensemble can be trained independently from the oth-ers). These improvements will be pursued in the fu-

2Based on current Google BigQuery pricing.

ture. Introduction of additional features will also beexplored to take into account in a more explicit man-ner the interaction between nodes. BigQuery will beused to extract these interactions and build networksto model them. Changes in the properties of thesenetworks over time could provide important informa-tion on possible future failures.

The method presented here is suitable for on-lineuse. A new model can be trained every day using thelast 12 days of logs. This is the scenario we simu-lated when we created the 15 test benchmarks. Themodel would be trained with 10 days of data andtested on the next non-overlapping day, exactly likein the benchmarks (Fig. 1). Then, it would be ap-plied for one day to predict future failures. The nextday a new model would be obtained from new data.Each time, only the last 12 days of data would beused rather than increasing the amount of trainingdata. This would account for the fact that the sys-tem itself and the workload can change over time, soold data may not be representative of current systembehavior. This would ensure that the model is up-to-date with the current system state. Testing on onenon-overlapping day is required for live use for tworeasons. First, part of the test data is used to buildthe ensemble (prediction-weighted voting). Secondly,the TPR and precision values on test data can helpsystem administrators make decisions on the critical-ity of the predicted failure when the model is applied.Here, we explored how simply putting the nodes thatare predicted to fail under quarantine can help re-cover many of the interrupted tasks and save some ofthe wasted resources. More sophisticated strategiescan be based on the criticality of predicted failures.

9 Acknowledgements

BigQuery analysis was carried out through a gener-ous Cloud Credits grant from Google. We are grate-ful to John Wilkes of Google for helpful discussionsregarding the cluster trace data.

15

References

[1] Omar Arif Abdul-Rahman and Kento Aida.Towards understanding the usage behavior ofGoogle cloud users: the mice and elephants phe-nomenon. In IEEE International Conferenceon Cloud Computing Technology and Science(CloudCom), pages 272–277, Singapore, Decem-ber 2014.

[2] Alkida Balliu, Dennis Olivetti, Ozalp Babaoglu,Moreno Marzolla, and Alina Sırbu. Bidal: Bigdata analyzer for cluster traces. In Informatika(BigSys workshop), volume 232, pages 1781–1795. GI-Edition Lecture Notes in Informatics,2014.

[3] Alkida Balliu, Dennis Olivetti, Ozalp Babaoglu,Moreno Marzolla, and Alina Sırbu. A big dataanalyzer for large trace logs. Computing, Inpress, 2015.

[4] David Breitgand, Zvi Dubitzky, Amir Epstein,Oshrit Feder, Alex Glikson, Inbar Shapira, andGiovanni Toffetti. An adaptive utilization accel-erator for virtualized environments. In Interna-tional Conference on Cloud Engineering (IC2E),pages 165–174, Boston, MA, USA, March 2014.IEEE.

[5] Faruk Caglar and Aniruddha Gokhale. iOver-book: intelligent resource-overbooking to sup-port soft real-time applications in the cloud.In 7th IEEE International Conference on CloudComputing (IEEE CLOUD), Anchorage, AK,USA, Jun–Jul 2014.

[6] Sheng Di, Derrick Kondo, and Walfredo Cirne.Characterization and Comparison of GoogleCloud Load versus Grids. In International Con-ference on Cluster Computing (IEEE CLUS-TER), pages 230–238, 2012.

[7] Sheng Di, Derrick Kondo, and Walfredo Cirne.Host load prediction in a Google compute cloudwith a Bayesian model. 2012 InternationalConference for High Performance Computing,Networking, Storage and Analysis, pages 1–11,November 2012.

[8] Sheng Di, Yves Robert, Frederic Vivien, Der-rick Kondo, Cho-Li Wang, and Franck Cap-pello. Optimization of cloud task processing withcheckpoint-restart mechanism. In 25th Interna-tional Conference on High Performance Com-puting, Networking, Storage and Analysis (SC),Denver, CO, USA, November 2013.

[9] Roman Dudko, Abhishek Sharma, and JonTedesco. Effective Failure Prediction in HadoopClusters. University of Idaho White Paper,pages 1–8, 2012.

[10] Ana Gainaru, Mohamed Slim Bouguerra, FranckCappello, Marc Snir, and William Kramer. Nav-igating the blue waters: Online failure predictionin the petascale era. Argonne National Labora-tory Technical Report, ANL/MCS-P5219-1014,2014.

[11] Mikel Galar, Alberto Fernandez, Edurne Bar-renechea, Humberto Bustince, and FranciscoHerrera. A review on ensembles for the classimbalance problem: bagging-, boosting-, andhybrid-based approaches. Systems, Man, andCybernetics, Part C: Applications and Reviews,IEEE Transactions on, 42(4):463–484, 2012.

[12] Qiang Guan and Song Fu. Adaptive anomalyidentification by exploring metric subspace incloud computing infrastructures. In 32ndIEEE Symposium on Reliable Distributed Sys-tems (SRDS), pages 205–214, Braga, Portugal,September 2013.

[13] Jesus Omana Iglesias, Liam Murphy Lero, Mi-lan De Cauwer, Deepak Mehta, and BarryO’Sullivan. A methodology for online consoli-dation of tasks through more accurate resourceestimations. In IEEE/ACM Intl. Conf. on Util-ity and Cloud Computing (UCC), London, UK,December 2014.

[14] Bahman Javadi, Derrick Kondo, Alexandru Io-sup, and Dick Epema. The Failure TraceArchive: Enabling the comparison of failuremeasurements and models of distributed sys-tems. Journal of Parallel and Distributed Com-puting, 73(8), 2013.

16

[15] Taghi M Khoshgoftaar, Moiz Golawala, and Ja-son Van Hulse. An empirical study of learningfrom imbalanced data using random forest. InTools with Artificial Intelligence, 2007. ICTAI2007. 19th IEEE International Conference on,volume 2, pages 310–317. IEEE, 2007.

[16] Ludmila I Kuncheva, Christopher J Whitaker,Catherine A Shipp, and Robert PW Duin. Isindependence good for combining classifiers? InPattern Recognition, 2000. Proceedings. 15th In-ternational Conference on, volume 2, pages 168–171. IEEE, 2000.

[17] Yinglung Liang, Yanyong Zhang, Hui Xiong,and Ramendra Sahoo. Failure Prediction inIBM BlueGene/L Event Logs. Seventh IEEE In-ternational Conference on Data Mining (ICDM2007), pages 583–588, October 2007.

[18] Zitao Liu and Sangyeun Cho. CharacterizingMachines and Workloads on a Google Cluster.In 8th International Workshop on Schedulingand Resource Management for Parallel and Dis-tributed Systems (SRMPDS), 2012.

[19] Asit K Mishra, Joseph L Hellerstein, WalfredoCirne, and Chita R. Das. Towards Character-izing Cloud Backend Workloads: Insights fromGoogle Compute Clusters. Sigmetrics perfor-mance evaluation review, 37(4):34–41, 2010.

[20] David W Opitz, Jude W Shavlik, et al. Gener-ating accurate and diverse members of a neural-network ensemble. Advances in neural informa-tion processing systems, pages 535–541, 1996.

[21] Charles Reiss, Alexey Tumanov, Gregory RGanger, Randy H Katz, and Michael A Kozuch.Heterogeneity and Dynamicity of Clouds atScale: Google Trace Analysis. In ACM Sym-posium on Cloud Computing (SoCC), 2012.

[22] Charles Reiss, Alexey Tumanov, Gregory RGanger, Randy H Katz, and Michael A Kozuch.Towards understanding heterogeneous clouds atscale: Google trace analysis. Carnegie MellonUniversity Technical Reports, ISTC-CC-TR(12-101), 2012.

[23] Charles Reiss, John Wilkes, and Joseph L Heller-stein. Obfuscatory obscanturism: making work-load traces of commercially-sensitive systemssafe to release. In Network Operations andManagement Symposium (NOMS), 2012 IEEE,pages 1279–1286. IEEE, 2012.

[24] Lior Rokach. Ensemble-based classifiers. Artifi-cial Intelligence Review, 33(1-2):1–39, 2010.

[25] A Rosa, LY Chen, Robert Birke, and W Binder.Demystifying Casualties of Evictions in Big DataPriority Scheduling. ACM SIGMETRICS Per-formance Evaluation Review, 42(4):12–21, 2015.

[26] Andrea Rosa, Lydia Y. Chen, and WalterBinder. Predicting and Mitigating Jobs Failuresin Big Data Clusters. In 2015 15th IEEE/ACMInternational Symposium on Cluster, Cloud andGrid Computing, pages 221–230. Ieee, May 2015.

[27] Felix Salfner, Maren Lenk, and Miroslaw Malek.A survey of online failure prediction methods.ACM Computing Surveys (CSUR), 42(3):1–68,2010.

[28] Taghrid Samak, Dan Gunter, Monte Goode,Ewa Deelman, Gideon Juve, Fabio Silva, andKaran Vahi. Failure analysis of distributed sci-entific workflows executing in the cloud. In Net-work and service management (cnsm), 2012 8thinternational conference and 2012 workshop onsystems virtualiztion management (svm), pages46–54. IEEE, 2012.

[29] Bikash Sharma, Victor Chudnovsky, Joseph LHellerstein, Rasekh Rifaat, and Chita R Das.Modeling and synthesizing task placement con-straints in google compute clusters. In Proceed-ings of the 2nd ACM Symposium on Cloud Com-puting, page 3. ACM, 2011.

[30] Catherine A. Shipp and Ludmila I. Kuncheva.Relationships between combination methodsand measures of diversity in combining classi-fiers. Information Fusion, 3(2):135 – 148, 2002.

17

[31] A. Sırbu and O. Babaoglu. Towards data-drivenautonomics in data centers. In Cloud and Auto-nomic Computing (ICCAC), 2015 InternationalConference on, pages 45–56, Sept 2015.

[32] Alina Sırbu and Ozalp Babaoglu. BigQueryand ML scripts. GitHub, 2015. Availableat https://github.com/alinasirbu/google_

cluster_failure_prediction.

[33] Jordan Tigani and Siddartha Naidu. Google Big-Query Analytics. John Wiley & Sons, 2014.

[34] Abhishek Verma, Luis Pedrosa, Madhukar R.Korupolu, David Oppenheimer, Eric Tune, andJohn Wilkes. Large-scale cluster managementat Google with Borg. In Proceedings of the Eu-ropean Conference on Computer Systems (Eu-roSys), Bordeaux, France, 2015.

[35] Guanying Wang, Ali R Butt, Henry Monti, andKaran Gupta. Towards Synthesizing Realis-tic Workload Traces for Studying the HadoopEcosystem. In 19th IEEE Annual InternationalSymposium on Modelling, Analysis, and Simu-lation of Computer and Telecommunication Sys-tems (MASCOTS), pages 400–408, 2011.

[36] John Wilkes. More Google cluster data.Google research blog, November 2011. Postedat http://googleresearch.blogspot.com/

2011/11/more-google-cluster-data.html.

[37] Qi Zhang, Joseph L Hellerstein, and RaoufBoutaba. Characterizing Task Usage Shapes inGoogle’s Compute Clusters. In Proceedings ofthe 5th International Workshop on Large ScaleDistributed Systems and Middleware, 2011.

[38] Qi Zhang, Mohamed Faten Zhani, RaoufBoutaba, and Joseph L Hellerstein. Dynamicheterogeneity-aware resource provisioning in thecloud. IEEE Transactions on Cloud Computing(TCC), 2(1), March 2014.

18