Computer Science and Artificial Intelligence Laboratory

Technical Report

m a s s a c h u s e t t s i n s t i t u t e o f t e c h n o l o g y, c a m b r i d g e , m a 0 213 9 u s a — w w w. c s a i l . m i t . e d u

MIT-CSAIL-TR-2014-004 March 24, 2014

Cicada: Predictive Guarantees for Cloud Network BandwidthKatrina LaCurts, Jeffrey C. Mogul, Hari Balakrishnan, and Yoshio Turner

Cicada: Predictive Guarantees for Cloud NetworkBandwidth

Katrina LaCurts*, Jeffrey C. Mogul†, Hari Balakrishnan*, Yoshio Turner‡*MIT CSAIL, †Google, Inc., ‡HP Labs

ABSTRACTIn cloud-computing systems, network-bandwidth guaranteeshave been shown to improve predictability of applicationperformance and cost [1, 30]. Most previous work on cloud-bandwidth guarantees has assumed that cloud tenants knowwhat bandwidth guarantees they want. However, applica-tion bandwidth demands can be complex and time-varying,and many tenants might lack sufficient information to re-quest a bandwidth guarantee that is well-matched to theirneeds. A tenant’s lack of accurate knowledge about its futurebandwidth demands can lead to over-provisioning (and thusreduced cost-efficiency) or under-provisioning (and thus pooruser experience in latency-sensitive user-facing applications).

We analyze traffic traces gathered over six months froman HP Cloud Services datacenter, finding that applicationbandwidth consumption is both time-varying and spatiallyinhomogeneous. This variability makes it hard to predictrequirements. To solve this problem, we develop a predictionalgorithm usable by a cloud provider to suggest an appro-priate bandwidth guarantee to a tenant. The key idea inthe prediction algorithm is to treat a set of previously ob-served traffic matrices as “experts” and learn online the bestweighted linear combination of these experts to make its pre-diction. With tenant VM placement using these predictiveguarantees, we find that the inter-rack network utilization incertain datacenter topologies can be more than doubled.

1. INTRODUCTION

This report introduces predictive guarantees, a new ab-straction for bandwidth guarantees in cloud networks. Apredictive guarantee improves application-performance pre-dictability for network-intensive applications, in terms ofexpected throughput, transfer completion time, or packet la-tency. A provider of predictive guarantees observes trafficalong the network paths between virtual machines (VMs),and uses those observations to predict the data rate require-ments over a future time interval. The provider can offer atenant a guarantee based on this prediction.

Why should clouds offer bandwidth guarantees, and whyshould they use predictive guarantees in particular? Cloudcomputing infrastructures, especially public “Infrastructure-as-a-Service” (IaaS) clouds, such as those offered by Amazon,

HP, Google, Microsoft, and others, are being used not justby small companies, but also by large enterprises. For dis-tributed applications involving significant network inter-nodecommunication, such as in [1], [24], and [25], current cloudsystems fail to offer even basic network performance guaran-tees; this inhibits cloud use by enterprises that must provideservice-level agreements (SLAs).

Others have proposed mechanisms to support cloud band-width guarantees (see §2.2); with few exceptions, these worksassume that tenants know what guarantee they want, and re-quire the tenants to explicitly specify bandwidth requirements,or to request a specific set of network resources. But do cloudcustomers really know their network requirements? Applica-tion bandwidth demands can be complex and time-varying,and not all application owners accurately know their band-width demands. A tenant’s lack of accurate knowledge aboutits future bandwidth demands can lead to over- or under-provisioning.

For many (but not all) cloud applications, future bandwidthrequirements are in fact predictable. In this report, we usenetwork traces from HP Cloud Services1 to demonstrate thattenant bandwidth demands can be time-varying and spatiallyinhomogeneous, but can also be predicted, based on auto-mated inference from their previous history.

We argue that predictive guarantees provide a better ab-straction than prior approaches, for three reasons. First, thepredictive guarantee abstraction is simpler for the tenant, be-cause the provider automatically predicts a suitable guaranteeand presents it to the tenant.

Second, the predictive guarantee abstraction supports time-varying and space-varying demands. Prior approaches typ-ically offer bandwidth guarantees that are static in at leastone of those respects, but these approaches do not capturegeneral cloud applications. For example, we expect temporalvariation for user-facing applications with diurnally-varyingworkloads, and spatial variation in VM-to-VM traffic forapplications such as three-tier services.

Third, the predictive guarantee abstraction easily supportsfine-grained guarantees. By generating guarantees automat-ically, rather than requiring the tenant to specify them, wecan feasibly support a different guarantee on each VM-to-

1http://hpcloud.com

1

1. customer makes initial request

2. provider places tenant VMs and sets initial rate limits 3. hypervisors continually send

measurements to Cicada controller

4. Cicada predicts bandwidth for each tenant

5a. provider may offer the customer a guarantee based

on Cicadaʼs prediction

hypervisors enforce rate limits and collect measurements

5b. provider may updateplacement based on Cicadaʼs prediction

Figure 1: Cicada’s architecture.

VM directed path, and for relatively short time intervals.Fine-grained guarantees are potentially more efficient thancoarser-grained guarantees, because they allow the providerto pack more tenants into the same infrastructure.

Recent research has addressed these issues in part. Forexample, Oktopus [1] supports a limited form of spatialvariation; Proteus [30] supports a limited form of temporal-variation prediction. But no prior work, to our knowledge,has offered a comprehensive framework for cloud customersand providers to agree on efficient network-bandwidth guar-antees, for applications with time-varying and space-varyingtraffic demands.

We place predictive guarantees in the concrete context ofCicada, a system that implements predictive guarantees. Ci-cada observes a tenant’s past network usage to predict itsfuture network usage, and acts on its predictions by offeringpredictive guarantees to the customer, as well as by placing(or migrating) VMs to increase utilization and improve loadbalancing in the provider’s network. Cicada is therefore ben-eficial to both the cloud provider and cloud tenants. Figure 1depicts Cicada’s architecture, which we describe in detail in§4.

Our primary contributions are the introduction of predic-tive guarantees, and trace-based analyses of the motivationfor, and utility of, our approach. We answer the followingquestions:

1. Do real applications exhibit bandwidth variability? Weshow that they do, using network traces from HP CloudServices, and thus justify the need for predictions basedon temporal and spatial variability in application traf-fic. While such variability is not surprising, it has notpreviously been quantified for cloud networks.

2. How well does Cicada predict network demands? Wedescribe Cicada’s prediction algorithm. The algorithmtreats certain previously observed traffic matrices as “ex-perts” and computes a weighted linear combination ofthe experts as the prediction. The weights are computedautomatically using online learning. Using our networktraces, we show that the median prediction errors of thealgorithm are 90% lower than other methods. Moreover,we can often predict when the prediction is likely tobe wrong (and avoid making guarantees in such cases).The algorithm can predict parameters for Oktopus’ Vir-tual Oversubscribed Cluster (VOC) model [1] nearly as

well as a perfect oracle.

3. Can a provider use Cicada’s predictions to better uti-lize its network infrastructure? We present a greedyVM-placement heuristic, which uses these predictions.For well-provisioned networks, a provider using thisheuristic can improve inter-rack bandwidth utilizationby over 2× in certain datacenter topologies, comparedto Oktopus.

2. RELATED WORK

We begin by putting Cicada in context of other, relatedsystems. We discuss related work on traffic prediction algo-rithms in §5.

2.1 DefinitionsWe use the following definitions. A provider refers to an

entity offering a public cloud (IaaS) service. A customer isan entity paying for use of the public cloud. We distinguishbetween customer and tenant, as tenant has a specific mean-ing in OpenStack.2: operationally “a collection of VMs thatcommunicate with each other.” A single customer may beresponsible for multiple tenants. Finally, application refersto software run by a tenant; a tenant may run multiple appli-cations at once.

2.2 Cloud-Network Bandwidth GuaranteesRecent research has proposed various forms of cloud net-

work guarantees. Oktopus supports a two-stage “virtual over-subscribed cluster” (VOC) model [1], intended to match a typ-ical application pattern in which clusters of VMs require highintra-cluster bandwidth and lower inter-cluster bandwidth.VOC is a hierarchical generalization of the hose model [6];the standard hose model, as used in MPLS, specifies foreach node its total ingress and egress bandwidths. The finer-grained pipe model specifies bandwidth values between eachpair of VMs. Cicada supports any of these models.

The Proteus system [30] profiles specific MapReduce jobsat a fine time scale, to exploit the predictable phased behaviorof these jobs. It supports a “temporally interleaved virtualcluster” model, in which multiple MapReduce jobs are sched-uled so that their network-intensive phases do not interferewith each other. Proteus assumes uniform all-to-all hose-model bandwidth requirements during network-intensive2http://docs.openstack.org/trunk/openstack-compute/admin/content/users-and-projects.html

2

phases, although each such phase can run at a different pre-dicted bandwidth. Unlike Cicada, it does not generalize to abroad range of enterprise applications.

Kim et al. describe a system that measures the time-varyingtraffic for a tenant’s VMs for a month, then uses a stable-marriage algorithm to place these VMs to reduce networkoversubscription [14]. They evaluated this scheme usingsynthetic traffic, showing an improvement over random orfirst-fit VM placement. Other recent work has focused onmaking traffic predictions to produce short-term (10-minute)guarantees for video streaming applications [20]. Althoughthis work considers VM-to-VM guarantees, it is not clearthat the approach generalizes to long-term guarantees, or toapplications beyond video streaming.

None of the papers discussed above, except for Proteus,address the problem of how the desired bandwidths are cho-sen. Hajjat et al. [10] describe a technique to decide whichapplication components to place in a cloud datacenter, forhybrid enterprises where some components remain in a pri-vate datacenter. Their technique tries to minimize the trafficbetween the private and cloud datacenters, and hence rec-ognizes that inter-component traffic demands are spatiallynon-uniform. In contrast to Cicada, they do not considertime-varying traffic nor how to predict it, and they focus pri-marily on the consequences of wide-area traffic, rather thanintra-datacenter traffic.

Cicada does not focus on the problem of enforcing guaran-tees. Though this issue would arise for a provider using Ci-cada, it can be addressed with numerous existing techniques.For example, SecondNet [9] supports either pipe-model orhose-model guarantees (their “type-0” and “type-1” services,respectively), and focuses on how to place VMs such thatthe guarantees are satisfied. Distributed Rate Limiting [23]supports a tenant-aggregate limit (similar to a hose model),and focuses on enforcing a limit at multiple sites, rather thanwithin one cloud datacenter. GateKeeper [26] provides a purehose-model guarantee, with the option of allowing additionalbest-effort bandwidth; its goal is to protect each VM’s input-bandwidth guarantee against adversarial best-effort traffic.NetShare [16] focuses on how to provide enforcement mech-anisms for cloud network guarantees; it may be viewed as agood way to achieve the “enforce rate limits” step of Figure 1.

Similarly, Cicada does not focus on the tradeoff betweenguarantees and fairness. Cloud providers could address thisissue by using an existing system such as FairCloud [22],which develops mechanisms that support various points inthe tradeoff space.

2.3 Datacenter Network Measurement StudiesDespite the importance of datacenter networks, the re-

search community has had scant access to measurements,because of the difficulty of gathering large-scale measure-ments, the privacy and security risks created by these datasets, and the proprietary value that providers place on un-derstanding what goes on in their networks. We know ofno published measurement studies on IaaS traffic matrices

(except perhaps [2], discussed below).Prior studies on similar networks have detected temporal

and spatial variability. Benson et al. [3] analyzed link-levelSNMP logs from 19 datacenters, finding “temporal and spa-tial variations in link loads and losses.” These, we believe,were not cloud (IaaS) networks per se (they may have beenSaaS/PaaS datacenters), although their applications may besimilar to those of cloud tenants. Benson et al. [2] gatheredSNMP statistics for ten datacenters and packet traces froma few switches in four datacenters. They describe several ofthese as “cloud data centers,” but it is unclear whether theyare actually IaaS networks. They report that “diurnal patterns[in link utilization] exist in all [ten] data centers,” and that“time-of-day and day-of-week variation exists in many of thedata centers,” especially in the cores of these networks.

Greenberg et al. [8] report on SNMP and NetFlow datafrom a “large cloud service provider.” We believe this, too, isnot an IaaS provider. They report a distinct lack of short-termpredictability for traffic matrices, but do not say whether thisdatacenter experiences diurnal variations. Kandula et al. [13]found considerable short-term spatial variation in a 1500-server data-mining cluster, but did not investigate whetherthis variation is predictable. Bodík et al. [4] also found spatialvariation in inter-service traffic in a datacenter, as a result ofthe fact that machines responsible for different services didnot typically communicate with one another.

2.4 Internet QoSQoS on the Internet was a vibrant research area for many

years, with architectures such as IntServ developed for end-to-end guarantees. End-to-end Internet QoS has seen littlepractical deployment, in part because most Internet paths in-volve multiple providers, making the economics and paymentstructure of any end-to-end guaranteed QoS scheme difficult.In contrast, a cloud network is run by a single operator, andinherently has a mechanism to bill its customers.

Another drawback of proposals such as IntServ is thatthey force the application, or end point, to make explicitreservations and to specify traffic characteristics. For allbut the simplest of applications, this task is challenging, andmany application developers or operators have little idea whattheir traffic looks like over time. Cicada resolves this issuethrough its use of predictions.

Cicada’s predictive approach resembles ISP traffic engi-neering, which ISPs deploy in practice. Traffic engineeringuses estimates of traffic to map flows or aggregates to pathsin a network. The primary difference is that Cicada makespredictions about the future, and uses these to offer predic-tive guarantees to tenants, and/or to place VMs so as to shifttraffic. Cicada does not deal with the problem of estimatingtraffic matrices from noisy link measurements, which manytraffic engineering schemes address.

3. MEASUREMENT RESULTS

We designed Cicada under the assumption that the trafficfrom cloud tenants is predictable, but in ways that are not

3

captured by existing models (e.g., VOC-style allocations [1],static, all-to-all traffic matrices, etc.). Before building Cicada,we collected data from HP Cloud Services, to analyze thespatial and temporal variability of its tenants’ traffic.

3.1 DataWe have collected sFlow [28] data from HP Cloud Services,

which we refer to as the HPCS dataset. Our dataset consistsof about six months of samples from 71 Top-of-Rack (ToR)switches. Each ToR switch connects to either 48 servers via1GbE NICs, or 16 servers via 10GbE NICs. In total, the datarepresents about 1360 servers, spanning several availabilityzones (portions of a datacenter that are isolated from failuresin other parts of the same datacenter). This dataset differsfrom those in previous work—such as [2, 3, 8]—in that itcaptures VM-to-VM traffic patterns. It does not include anyinformation about what types of applications were running(e.g., MapReduce jobs, webservers, etc.), as that informationis not available from packet headers.

Under the agreement by which we obtained this data, weare unable to reveal information such as the total number oftenants, the number of VMs per tenant, or the growth ratesfor these values.

3.1.1 Dataset LimitationsDuring data collection, the physical network was generally

over-provisioned. Hence, our measurements reflect the actualoffered loads at the virtual interfaces of the tenant VMs. Someof the smaller VMs were output-rate-limited at their virtualinterfaces by HPCS; we do not know these rate limits.

Because the HPCS dataset samples come from the ToRswitches, they do not include any traffic between pairs of VMswhen both are on the same server. We believe that we stillget samples from most VM pairs, because, in the measuredconfiguration (OpenStack Diablo), VMs are placed on serversuniformly at random. Assuming such placement on n servers,the probability of any two of a tenant’s k VMs being on thesame server is 1/n; with n = 1360, the probability that we willmiss any given VM-pair’s traffic is less than .001%.3

We aggregate the data over five-minute intervals, to pro-duce datapoints of the form 〈timestamp, source, destination,number of bytes transferred from source to destination〉. Wekeep only the datapoints where both the source and desti-nation are private IPs of virtual machines, and thus all ourdatapoints represent traffic within the datacenter. Makingpredictions about traffic traveling outside the datacenter orbetween datacenters is a challenging task, which we do notaddress in this work.

For privacy reasons, our dataset does not include informa-tion associating VMs with tenants or applications. Instead,we define tenants by using the connected components ofthe full traffic matrix, which is in line with the OpenStackdefinition of tenants.3The expected number of unordered pairs of k VMs sharing n serversis 1

n(k

2). There are

(k2)

possible unordered pairs among k VMs, sothe probability that any VM-pair shares the same server is 1

n (k2)/(k

2).

0

0.2

0.4

0.6

0.8

1

0.01 0.1 1 10 100

CD

F

Spatial Variation

Figure 2: Spatial variation in the HPCS dataset.

3.1.2 Measurement-Interval GranularityCicada’s predictive guarantees are parameterized on two

values: H, the amount of time the guarantee is good for,and δ , the peak-bandwidth measurement interval. Duringa period of H hours, Cicada’s prediction reflects what thepeak-bandwidth should be in any δ -second interval (e.g., ifH = 1 and δ = 300, Cicada’s prediction reflects the maximumamount of bandwidth that it expects the application to usein any 300-second interval for the next hour). For latency-sensitive applications, predictions for δ << H are useful; forlatency-insensitive applications, δ ≈ H.

Because our data collection samples samples over 5-minuteintervals, for our datasets, δ ≥ 300 seconds. Real applicationsmight require guarantees with δ < 1 second; our implemen-tation supports these small δ values (§8.1).

3.1.3 Over-samplingSome of the paths in the HPCS dataset are over-sampled.

If A and B are VMs on the same ToR switch S1, traffic onA ; B will only be sampled at S1. But if VM C resides on adifferent ToR S2, traffic on A ; C will be sampled at both S1and S2, twice as often as A ; B.

We correct this problem by noting which switches we seeon each path (sFlow samples are tagged with the ToR’s ownaddress). If we see two switches on a path, we know that thisflow has been oversampled, so we scale those measurementsdown by a factor of two.

3.2 Spatial VariabilityTo quantify spatial variability, we compare the observed

tenants to an ideal, static, all-to-all tenant (this tenant is idealas it is the easiest to make predictions for: every intra-tenantconnection uses the same amount of bandwidth all of thetime). Let Fi j be the fraction of this tenant’s traffic sent fromV Mi to V M j. For the “ideal” all-to-all tenant, the distributionof these F values has a standard deviation of zero, since everyVM sends the same amount of data to every other VM.

For each tenant in the HPCS dataset, we calculate the dis-tribution of its F values, and plot the coefficient of variation(standard deviation over mean) in Figure 2. The mediancov value is 1.732, which suggests nontrivial overall spatialvariation.

4

0

0.2

0.4

0.6

0.8

1

0.001 0.01 0.1 1 10 100

CD

F

Temporal Variation

1 hour

4 hours

24 hours

Figure 3: Temporal variation in the HPCS dataset.

Some VMs communicate much more than others: onetenant with cov > 10 has eight VMs, with a few pairs thatsend modest amounts of data, and all other pairs sending littleto no data. These results suggest that a uniform, all-to-allmodel is insufficient for making traffic predictions; giventhe high cov values in Figure 2, a less strict but not entirelygeneral model such as VOC [1] may not be sufficient either.

3.3 Temporal VariabilityTo quantify temporal variability, we first pick a time in-

terval H. For each consecutive non-overlapping interval ofH hours, we calculate the sum of the total number of bytessent between each pair p of VMs. This gives us a distributionTp of bandwidth totals. We then compute the coefficient ofvariation for this distribution, covp. The temporal variationfor a tenant is the weighted sum of these values, where theweight for covp is the bandwidth used by pair p. This scalingreflects the notion that tenants where only one small flowchanges over time are less temporally-variable than thosewhere one large flow changes over time.

For each tenant in the HPCS dataset, we calculate its tem-poral variation value, and plot the CDF in Figure 3. Like thespatial variation graph, Figure 3 shows that most tenants havehigh temporal variability. This variability decreases as weincrease the time interval H, but we see variability at all timescales. Tenants with high temporal variation are typicallyones that transfer little to no data for long stretches of time,interspersed with short bursts of activity.

The results so far indicate that typical cloud tenants maynot fit a rigid model. In particular, Figure 3 shows that mosttenants exhibit significant temporal variation. Thus, staticmodels cannot accurately represent the traffic patterns of thetenants in the HPCS dataset.

4. THE DESIGN OF CICADA

The goal of Cicada is to free tenants from choosing be-tween under-provisioning for peak periods, or over-paying forunused bandwidth. Predictive guarantees permit a providerand tenant to agree on a guarantee that varies in time and/orspace. The tenant can get the network service it needs at agood price, while the provider can avoid allocating unneededbandwidth and can amortize its infrastructure across more

tenants.

4.1 An Overview of CicadaCicada has several components, corresponding to the steps

in Figure 1. After determining whether to admit a tenant—taking CPU, memory, and network resources into account—and making an initial placement (steps 1 and 2), Cicada mea-sures the tenant’s traffic (step 3), and delivers a time series oftraffic matrices to a logically centralized controller. The con-troller uses these measurements to predict future bandwidthrequirements (step 4). In most cases, Cicada converts a band-width prediction into an offered guarantee for some futureinterval. Customers may choose to accept or reject Cicada’spredictive guarantees (step 5). The customer might also pro-pose its own guarantee, for which the provider can offer aprice. Because Cicada collects measurement data continu-ally, it can make new predictions and offer new guaranteesthroughout the lifetime of the tenant.

Cicada interacts with other aspects of the provider’s infras-tructure and control system. The provider needs to rate-limitthe tenant’s traffic to ensure that no tenant undermines theguarantees sold to other customers. We distinguish betweenguarantees and limits. If the provider’s limit is larger thanthe corresponding guarantee, tenants can exploit best-effortbandwidth beyond their guarantees.

A provider may wish to place and perhaps migrate VMsbased on their associated bandwidth guarantees, to improvenetwork utilization. We describe a VM migration method in§7.4. The provider could also migrate VMs to increase thenumber of guarantees that the network can support [7].

4.2 AssumptionsIn order to make predictions, Cicada needs to gather suffi-

cient data about a tenant’s behavior. Based on our evaluation,Cicada may need at least an hour or two of data before it canoffer useful predictions.

Any shared resource that provides guarantees must includean admission control mechanism, to avoid making infeasibleguarantees. We assume that Cicada will incorporate net-work admission control using an existing mechanism, suchas [1], [12], or [15]. We also assume that the cloud providerhas a method to enforce guarantees, such as [22].

4.3 Measurement CollectionCicada collects a time series of traffic matrices for each

tenant. One could do this passively, by collecting NetFlowor sFlow data at switches within the network, or by using anagent that runs in the hypervisor of each machine. Switch-based measurements create several challenges, including cor-rectly ascribing VMs to the correct tenant. Our design collectsVM-pair traffic measurements, using an agent that runs oneach compute server (see §8.1), and periodically reports theseto the controller.

We would like to base predictions on offered load, butwhen the provider imposes rate limits, we risk underesti-mating peak loads that exceed those limits, and thus under-predicting future traffic. We observe that we can detect when

5

a VM’s traffic is rate-limited (see §8.1), so these underesti-mates can be detected, too, although not precisely quantified.Currently, Cicada does not account for this potential error.

4.4 Prediction ModelSome applications, such as backup or database ingestion,

require bulk bandwidth—that is, they need guarantees thatthe average bandwidth over a period of H hours will meettheir needs. Other applications, such as user-facing systems,require guarantees for peak bandwidth over much shorterintervals. Thus, Cicada’s predictions describe the maximumbandwidth expected during any averaging interval δ duringa given time interval H. If δ = H, the prediction is for thebandwidth requirement averaged over H hours, but for aninteractive application, δ might be just a few milliseconds.Our current implementation produces a prediction once perhour (H = 1).

Note, of course, that the predictive guarantees offered byCicada are limited by any caps set by the provider; thus, aproposed guarantee might be lower than suggested by theprediction algorithm.

A predictive guarantee entails some risk of either under-provisioning or over-provisioning, and different tenants willhave different tolerances for these risks, typically expressedas a percentile (e.g., the tenant wants sufficient bandwidth for99.99% of the 10-second intervals). Cicada uses the resultsof the prediction to determine whether it can issue reliablepredictive guarantees for a tenant; if not, it does not proposesuch a guarantee.

4.5 Recovering from Faulty PredictionsCicada’s prediction algorithm may make faulty predictions

because of inherent limitations or insufficient prior informa-tion. Because Cicada continually collects measurements, itcan detect when its current guarantee is inappropriate for thetenant’s current network load.

When Cicada detects a faulty prediction, it can take one ofmany actions: stick to the existing guarantee, propose a newguarantee, upgrade the tenant to a higher, more expensiveguarantee (and perhaps bear some fraction of the cost of theupgrade), etc. How and whether to upgrade guarantees, aswell as what to do if Cicada over-predicts, is a pricing-policydecision, and outside our scope. We note, however, thattypical System Level Agreements (SLAs) include penaltyclauses, in which the provider agrees to remit some or all ofthe customer’s payment if the SLA is not met.

We also note that a Cicada-based provider must maintainthe trust of its customers: it cannot regularly under-predictbandwidth demands, or else tenants will have insufficientguarantees and their own revenues may suffer; it also cannotregularly over-predict demands, or else customers will beover-charged and take their business elsewhere. This moti-vates our focus in §7 on evaluating the quality of Cicada’spredictions, including its decision whether a tenant’s demandsare in fact predictable.

5. CICADA’S TRAFFIC PREDICTION METHOD

Much work has been done on predicting traffic matricesfrom noisy measurements such as link-load data [27, 31].In these scenarios, prediction approaches such as Kalmanfilters and Hidden Markov Models—which try to estimatetrue values from noisy samples—are appropriate. Cicada,however, knows the exact traffic matrices observed in pastepochs (an epoch is H-hours long; typically one hour); itsproblem is to predict a future traffic matrix.

To the best of our knowledge, there is little work in thisarea, especially in the context of cloud computing. On ISPnetworks, COPE [29] describes a strategy that predicts thebest routing over a space of traffic predictions, made upof the convex hull of previous traffic matrices. It is not aprediction scheme per se, but we draw inspiration from thiswork, and base our prediction algorithm around computing a“best” convex combination of previous traffic matrices as ourfuture estimate.

5.1 AlgorithmThe algorithm uses Herbster and Warmuth’s “tracking the

best expert” idea [11], which has been successfully adaptedbefore in wireless power-saving and energy reduction con-texts [5, 18]. To predict the traffic matrix for epoch n+1, weuse all previously observed traffic matrices, M1, . . . ,Mn (later,we show that matrices from the distant past can be prunedaway without affecting accuracy). In our context, the rowsand columns of the matrix are each VMs, with the entry inrow i and column j specifying the number of bytes that VM isent to VM j in the corresponding epoch (for predicting av-erage bandwidth) or the maximum observed over a δ -lengthinterval (for peak bandwidth).

Each of these previously observed matrices acts as an “ex-pert,” recommending that Mi is the best predictor, M̂n+1, forepoch n + 1. The algorithm computes M̂n+1 as a weightedlinear combination of these matrices:

M̂n+1 =n

∑i=1

wi(n) ·Mi

where wi(n) denotes the weight given to Mi when making aprediction for epoch n+1, with ∑n

i=1 wi(n) = 1.The algorithm learns the weights online. At each step, it

updates the weights according to the following rule:

wi(n+1) =1

Zn+1·wi(n) · e−L(i,n)

where L(i,n) denotes the loss of expert i in epoch n and Zn+1normalizes the distribution so that the weights sum to unity.

We use the relative Euclidean `2-norm between Mi and Mnas the loss function L. Treating both these matrices as vectors,~Mi and ~Mn respectively, the relative `2-norm error is

L(i,n) = E`2 = ‖~Mi− ~Mn‖/‖ ~Mn‖.

That is, the norm of the individual errors over the Euclideannorm of the observed data (the square-root of the sum of thesquares of the components of the vector).

6

0

0.1

0.2

0.3

0.4

0.5

-48 -24 -12 -1

Weig

ht

Index(an index of -x indicates that the matrix is from x hours ago)

Figure 4: Average weights produced by Cicada’s predic-tion algorithm. These values represent the average of thefinal weight values over each application in the HPCSdataset.

Note that this algorithm can predict both average band-width as well as peak bandwidth. The only difference is the in-put matrices: for average bandwidth, the matrices M1, . . . ,Mnrepresent the total amount of data in each epoch, while forpeak bandwidth, they represent the maximum over a δ -lengthinterval. It is also easy to extend this model to produce hose-model predictions rather than pipe-model predictions, simplyby using input matrices that represent the hose bandwidth(in this case, each matrix is a 1×n matrix, and entry i corre-sponds to the number of bytes sent out of VM i).

5.1.1 IntuitionOur intuition when approaching the problem of predicting

traffic is that the traffic matrix Mn should depend most heavilyon the most recent traffic matrices (Mn−1,Mn−2, etc.), as wellas on traffic matrices from similar time periods on previousdays (Mn−24,Mn−48, etc.).

If our intuition were correct, Cicada’s prediction algorithmwill naturally result in higher weights for these matrices. Af-ter making the predictions for each application in our dataset,we took the weights for each application, and calculated theaverage weight values over our entire dataset. These averageweights are plotted in Figure 4 (the x axis is limited to thetwo most recent days of data). The 12 most recent hours oftraffic are weighted heavily, and there is also a spike at 24hours earlier. Weights are vanishingly small prior to 24 hoursearlier. In particular, we looked for a spike at the 7-day offset,expecting that some user-facing applications have weeklyvariations, but found none. This result indicates that, at leastin our dataset, one does not need weeks’ worth of data tomake reliable predictions; a much smaller amount of datasuffices.

5.2 Alternate Prediction AlgorithmsWe tested Cicada’s prediction algorithm against two other

algorithms. First, we tried a machine learning algorithmbased on linear regression. A prediction between VMs i andj was made by finding “relevant” historical data between i andj, finding the linear function f that best mapped a previous

epoch’s data to the next epoch’s, and using f to make aprediction for the current time. The “relevant” historicaldata was data between i and j from similar times-of-dayand days-of-week; similarity was determined via a Mann-Whitney U-Test, as in [17]. This algorithm was also based onour intuition that traffic from the same time-of-day would bemore relevant than traffic from other times. In practice, wefound that this algorithm performed comparably to the bank-of-experts-based algorithm on average bandwidth predictions,but was very unreliable for peak bandwidth predictions; forthat reason, we do not present detailed results in §7.

We also tested Cicada’s algorithm against a simple EWMA.This technique performed worse than Cicada’s algorithm forboth peak and average bandwidth prediction, and so we donot present detailed results in §7.

6. COMPARING CICADA TO VOCBecause we believe that the VOC model [1] represents the

state of the art in spatially-varying cloud-bandwidth reser-vations, in our trace-based evaluation of Cicada, we use aVOC-style system as the baseline. In [1], the authors useVOCs to make bandwidth reservations for tenants, allowingdifferent levels of bandwidth between different groups ofVMs. Although their system was not developed for makingbandwidth predictions, we can interpret its bandwidth reser-vations as predictions. We compare a VOC-style system toCicada, finding that Cicada can accurately predict the param-eters for a VOC-style system, and that Cicada’s pipe modelresults in less waste than VOC’s hierarchical hose model.

6.1 VOC-style GuaranteesThe Oktopus system introduced the Virtual Oversubscribed

Cluster (VOC) model, to “capitalize on application structureto reduce the bandwidth needed from the underlying physicalinfrastructure” [1]. In VOC, the tenant’s guarantee request isof the form 〈N,B,S,O〉, such that a virtual network of N VMsis divided into groups of size S. The VMs within each groupare effectively connected to each other via links of bandwidthB through a non-oversubscribed virtual switch. Between anypair of groups, VOC provides bandwidth B/O; thus, O is theeffective oversubscription factor for inter-group traffic. VOCis designed to reflect not only a typical application structure,but also a typical physical datacenter topology, where Top-of-Rack (ToR) switches have high capacity, but the aggregationlayer that connects the ToRs is oversubscribed.

Oktopus itself is designed to place the VMs on the networksuch that their VOC requests are met. It assumes that thetenants are responsible for choosing the parameters 〈B,S,O〉.We designed a heuristic, detailed in the Appendix, to deter-mine these parameters, and to allow variable cluster sizes.

To compare Cicada and VOC, we interpret the B and B/Ohoses as predictions: VOC predicts that a VM in a particulargroup will use B bandwidth to send to VMs within its cluster,and groups will use B/O bandwidth to send to other groups.These predictions are not pipe-model predictions, as is ourpreferred model for Cicada, but we are still able to compare

7

vm1

vm2 vm3

10 10

1 1

1

1

(a) This traffic pattern will re-sult in over-allocation withinone VOC group.

100

group2 group3

group1

10010

10

10

10

(b) Similarly, this traffic pat-tern results in over-allocationacross VOC groups.

Figure 5: Cases where VOC is inefficient.

the accuracy of the two systems on a tenant-by-tenant basis.

6.2 Inefficiencies of VOCThe Oktopus paper showed that the VOC model gives

providers more flexibility than a clique abstraction, whichprovides a static, uniform bandwidth between all pairs ofVMs. However, we give an example to illustrate how eventhe VOC model can limit provider flexibility for certain appli-cations, due to over-allocating bandwidth both within groupsand across groups.

Consider a group of three VMs as in Figure 5(a). Supposethat V M1 sends 20 units total to V M2 and V M3. Becauseof this, B must be at least 20 for each VM in the group.However, if V M2 and V M3 send fewer than 20 units total, thisvalue of B will over-allocate bandwidth. In practical terms,if each VM is on a distinct server, the VOC model requiresallocating 20 units of each server’s NIC output bandwidthto this tenant, even though V M2 and V M3 only need 2 NIC-output units. A similar pattern can exist across groups, as inFigure 5(b), where one group requires more total bandwidththan the others.

VOC’s over-allocation of bandwidth in these scenariosstems from an assumption that VMs within a group behavesimilarly (sending the same amount of data to the rest of thegroup), as well as a corresponding assumption across groups.

7. EVALUATION

We evaluated our data on the HPCS dataset. We testedseveral hypotheses: first, that Cicada can determine when oneof its predictions is reliable or not; second, that Cicada canaccurately predict a tenant’s future bandwidth requirements;and third, that predictions incorporating spatial and temporalvariation can yield guarantees that waste less bandwidth thanuniform guarantees or static. We define “wasted bandwidth”as the bandwidth that is guaranteed to a tenant but not used.Wasted bandwidth is a proxy for estimating how much moneya customer would save—methods that waste less bandwidthwill likely save the customers money—but allows us to avoiddefining a particular cost model. Overall, we found that thereliability of Cicada’s predictions was correlated with thesize of a tenant and the frequency of under-predictions, thatCicada’s guarantees were accurate for both average and peakpredictions, and that Cicada’s guarantees were more accurate

-1

0

1

2

3

4

5

0 24 48 72 96 120 144

Rela

tive E

rro

r

Number of Hours of Historical Data

Figure 6: Relative error vs. history

than guarantees based on VOC, decreasing the relative per-tenant error by 90% in both the average-bandwidth case andthe peak-bandwidth case.

We observed very little evidence of VM flexing in ourdataset (where the number of a tenant’s VMs changes overtime). Flexing would typically manifest as over-predictionerrors (making predictions for VM pairs that used to exist, butno longer do because of flexing). To eliminate this mistake,we eliminated any predictions where the ground truth datawas zero, but found that it did not appreciably change theresults. For this reason, we do not present separate results inthis section, and simply report the results over all of the data.

7.1 Quantifying Prediction AccuracyTo quantify the accuracy of a prediction, we compare the

predicted values to the ground truth values, using the relative`2-norm error described in §5.1. For Cicada, the predictionand ground-truth vectors are of length N2−N, because Ci-cada makes predictions between each pair of distinct VMs. Ina VOC-style system, there are fewer predictions: one predic-tion for each of the N VMs to the other VMs in its group, andone prediction for each group to the other groups. Regardlessof the number of total predictions, we get one error value pertenant, per time interval.

In addition to the relative `2-norm error, we also present theper-tenant relative error. This metric is simply the sum of theprediction errors divided by the total amount of ground-truthdata. Unlike the relative `2-norm error, this metric discrim-inates between over-prediction and under-prediction, sincethe latter can be more disruptive to application performance.Because it does not use vector norms, the per-tenant relativeerror makes it easier for us to see if either system has substan-tially under-predicted for a tenant. However, it is possiblethat under- and over-prediction errors for different VM pairscould cancel out in the relative error; this type of cancellationdoes not happen in the relative `2-norm error.

7.2 Determining Whether Predictions Are ReliableIn our dataset, we found that Cicada could not reliably

make a correct prediction for tenants with few VMs. In allof the results that follow, we eliminate tenants with fewerthan five VMs. This elimination is in line with the fact that

8

0

0.2

0.4

0.6

0.8

1

0 1 2 3 4 5

CD

F

Relative L2-norm Error

Cicada (Pipe-model)Cicada (Hose-model)

VOC (Oracle)VOC (Predictions)

0

0.2

0.4

0.6

0.8

1

-1 0 1 2 3 4 5

CD

F

Relative Error

Cicada (Pipe-model)Cicada (Hose-model)

VOC (Oracle)VOC (Predictions)

Figure 7: Prediction errors for average bandwidth.4

Cicada is meant for tenants with network-heavy workloads.It is possible that on other datasets, the precise number oftenants below which Cicada cannot reliably make predictionswill differ. Additionally, for tenants that started out with aseries of under-predictions, Cicada’s prediction algorithmrarely recovered. For this reason, we also consider tenantswith more than fifteen under-predictions to be unreliable, anddo not continue making predictions for them (we do, however,include results from all predictions for that tenant up to thatpoint).

Interestingly, we found that Cicada could often make accu-rate predictions even with very little historical data. Figure 6shows the relative per-tenant error as a function of the amountof historical data. Though there is a clear correlation betweenrelative error and the amount of historical data, it is not nec-essary to have many hours of data in order to make accuratepredictions.

7.3 Prediction Errors for Individual TenantsTo compare the accuracy of Cicada and a VOC-style model,

we make predictions for one-hour intervals. We allow bothtypes of guarantees to change over time; that is, the VOCconfiguration for a particular hour need not be the same as theconfiguration in the next hour. This is an extension from theoriginal Oktopus paper [1], and improves VOC’s performancein our comparison.

We evaluate VOC-style guarantees using both predictedparameters and “oracle-generated” parameters (i.e., with per-

4For average bandwidths, straightforward math shows that Cicada’shose-model and pipe-model relative errors will always be identical.The peak-bandwidth errors, shown in Figure 8, can diverge.

fect hindsight). For the oracle parameters, we determine theVOC clusters for a prediction interval using the ground-truthdata from that interval. This method allows us to select theabsolute best values for B and O for that interval, as well asthe best configuration. Thus, any “error” in the VOC-oracleresults comes from the constraints of the model itself, ratherthan from an error in the predictions.

7.3.1 Average Bandwidth GuaranteesFigure 7 shows the results for average bandwidth predic-

tion. Both Cicada models have lower error than either VOCmodel; Cicada’s pipe model decreases the error by 90% com-pared to the VOC oracle model (comparison against the pre-dictive VOC model, as well as between VOC and Cicada’shose model, yields a similar improvement). The `2-normerror decreases by 71%. The VOC model using predictionsalso closely tracks the VOC-oracle model, indicating that Ci-cada’s prediction algorithm can generate accurate predictionsfor a system using VOC.

In terms of per-tenant relative error, Cicada occasionallyunder-predicts, whereas neither VOC model does. Under-prediction could be worse than over-prediction, as it meansthat an application’s performance could be reduced. Theeffect of under-provisioning can be lessened by scaling pre-dictions by an additive or multiplicative factor, though thisrisks over-prediction. In the results presented here, we havescaled the predictions by 1.25×. In addition to lesseningthe effects of under-provisioning, this scaling also allowsthe bank-of-experts algorithm to make a prediction that isgreater than any of the previous matrices. We were unable todetermine a systematic way to remove the remaining under-predictions, but speculate that altering the loss function topenalize under-prediction more heavily than over-predictionsmay help; we leave this to future work.

7.3.2 Peak Bandwidth GuaranteesFigure 8 compares peak predictions for δ = 5min. As

with the average-bandwidth predictions, Cicada’s predictionerrors are generally lower, decreasing the median error againby 90% from the VOC oracle model (the median `2-normerror decreases by 80%). As before, Cicada does under-predict more frequently than either VOC model, but overall,the results for peak-bandwidth guarantees show that Cicadaperforms well even for non-average-case traffic.

7.3.3 DiscussionThough Cicada’s prediction algorithm performs well on

our dataset, we make no claim that it is the best predictionalgorithm. Rather, we have shown that accurate traffic pre-diction is possible and worthwhile in the context of cloudcomputing.

7.4 Placement for Reducing Intra-rack BandwidthIn the previous sections, we reported wasted bandwidth

as a total; however, it is not clear that cloud providers treatwasted intra-rack and inter-rack bandwidth equally. Inter-rackbandwidth may cost more, and even if intra-rack bandwidth

9

0

0.2

0.4

0.6

0.8

1

0 1 2 3 4 5

CD

F

Relative L2-norm Error

Cicada (Pipe-Model)Cicada (Hose-Model)

VOC (Oracle)VOC (Predictions)

0

0.2

0.4

0.6

0.8

1

-1 0 1 2 3 4 5

CD

F

Relative Error

Cicada (Pipe-Model)Cicada (Hose-Model)

VOC (Oracle)VOC (Predictions)

Figure 8: Prediction errors for peak bandwidth.

Algorithm 1 Cicada’s VM placement algorithm1: P = set of VM pairs, in descending order of their band-

width predictions2: for (src,dst) ∈ P do3: if resources aren’t available to place (src,dst) then4: revert reservation5: return False6: A = All available paths7: if src or dst has already been placed then8: restrict A to paths including the appropriate endpoint9: Place (src,dst) on the path in A with the most available

bandwidth

is free, over-allocating network resources on one rack canprevent other tenants from being placed on the same rack(due to a presumed lack of network resources).

To determine whether wasted bandwidth is intra- or inter-rack, we need a VM-placement algorithm. For VOC, we usethe placement algorithm detailed in [1], which tries to placeclusters on the smallest subtree that will contain them. ForCicada, we developed a greedy placement algorithm, detailedin Algorithm 1. This algorithm is similar in spirit to VOC’splacement algorithm; Cicada’s algorithm tries to place themost-used VM pairs on the highest-bandwidth paths, whichin a typical datacenter corresponds to placing them on thesame rack, and then the same subtree. However, since Cicadauses fine-grained, pipe-model predictions, it has the potentialto allocate more flexibly; VMs that do not transfer data toone another need not be placed on the same subtree, even ifthey belong to the same tenant.

We compare Cicada’s placement algorithm against VOC’s,

0

100000

200000

300000

400000

500000

600000

1 1.4142 1.5 1.666 1.75 2 4 8 16

Inte

r-ra

ck B

an

dw

idth

Availab

le(G

Byte

/ho

ur)

Oversubscription Factor

bw factor 1x

bw factor 25x

bw factor 250x

Figure 9: Inter-rack bandwidth available after placingapplications.

on a simulated physical infrastructure with 71 racks with16 servers each, 10 VM slots per server, and (10G/Op)Gbpsinter-rack links, where Op is the physical oversubscriptionfactor. For each algorithm, we select a random tenant, anduse the ground truth data to determine this tenant’s bandwidthneeds for a random hour of its activity, and place its VMs. Werepeat this process until 99% of the VM slots are filled. Usingthe ground-truth data allows us to compare the placementalgorithms explicitly, without conflating this comparison withprediction errors. To get a sense of what would happen withmore network-intensive tenants, we also evaluated scenarioswhere each VM-pair’s relative bandwidth use was multipliedby a constant bandwidth factor (1×, 25×, or 250×).

Figure 9 shows how the available inter-rack bandwidth—what remains unallocated after the VMs are placed—varieswith Op, the physical oversubscription factor. In all cases, Ci-cada’s placement algorithm leaves more inter-rack bandwidthavailable. When Op is greater than two, both algorithmsperform comparably, since this is a constrained environmentwith little bandwidth available overall. However, with lowerover-subscription factors, Cicada’s algorithm leaves morethan twice as much bandwidth available, suggesting that ituses network resources more efficiently in this setting.

Over-provisioning reduces the value of our improved place-ment, but it does not necessarily remove the need for betterguarantees. Even on an over-provisioned network, a tenantwhose guarantee is too low for its needs may suffer if its VMplacement is unlucky. A “full bisection bandwidth” networkis only that under optimal routing; bad routing decisions orbad placement can still waste bandwidth.

8. IMPLEMENTATION

We have designed an implementation of Cicada as partof the OpenStack [21] framework. The implementation hastwo components: rate-limiters, which run in the hypervisorof every physical machine, and a controller that runs some-where in the datacenter. We envision that Cicada extends theOpenStack API to allow the tenant’s application to automati-cally negotiate its network guarantees, rather than requiringhuman interaction. A typical setting might be “Accept all

10

of Cicada’s guarantees as long as my bill does not exceedD dollars,” or “Accept all of Cicada’s guarantees, but adda buffer of 10Mb/s to each VM-pair’s allocated bandwidth”(if the customer anticipates some amount of unpredictabletraffic).

8.1 Server AgentsOn each server, Cicada uses an agent process to collect

traffic measurements and to manage the enforcement of ratelimits. The agent is a Python script, which manages theLinux tc module in the hypervisor. Cicada’s agent uses tcboth to count packets and bytes for each VM-pair with asender on the server, and, via the the HTB queuing discipline,to limit the traffic sent on each pair in accordance with thecurrent guarantee-based allocation. This type of collectionis more fine-grained than sFlow, and tc also allows us todetect, but not precisely quantify, traffic demands in excess ofthe rate limit, by counting packet drops. The agent receivesmessages from the controller that provide a list of rate limits,for the 〈src,dst〉 pairs where src resides on that server. It alsoaggregates counter values for those pairs, and periodicallyreports them to the controller. To avoid overloading thecontroller, the reporting period P is larger than the peak-measurement period δ .

8.2 Centralized ControllerThe centralized controller is divided into two components.

The prediction engine receives and stores the data about thetenants from each server agent. Once a prediction is madeand approved by the tenant, the rate controller uses the Open-Stack API to determine which VMs reside on which physicalserver, and communicates the relevant rates to each rate lim-iter. At this point, Cicada could also use its placement algo-rithm (Algorithm 1) to determine whether any VMs shouldbe migrated, and migrate them via the OpenStack API. SinceCicada makes long-term traffic predictions, this migrationwould be done at long timescales, mitigating the overhead ofmigrating VMs.

8.3 ScalabilityWhile our current controller design is centralized, it does

not need to be. Since Cicada makes predictions about ap-plications individually, the prediction engine can be spreadacross multiple controllers, as long as all of the data for aparticular application is accessible by the controller assignedto that application.

Cicada’s use of pipe-model rate limiting—that is, one rate-limiter for each VM-pair, at the source hypervisor—couldpotentially create scaling problems. Some prior work has alsoused pipe-model rate limiting; for example, while Oktopusimplements hose-model guarantees, it enforces these using“per-destination-VM rate limiters” [1]. However, we havefound no published study of software rate-limiter scaling.

We tested the scalability of the tc rate-limiters used in ourimplementation on a pair of 12-core Xeon X5650 (2.67GHz)servers running Linux 3.2.0-23, with 10Gbps NICs. Weused netperf to measure sender-side CPU utilization for

0

1

2

3

4

5

6

7

8

9

1 4 16 64 256 1024 4096

Se

nd

er

CP

U U

tili

za

tio

n (

%)

Number of Active Rate Limiters

100Mbps500Mbps1Gbps5Gbps10GbpsLow-BW

Figure 10: Effect of rate limiters on CPU utilization.

60-second TCP transfers under various rate limits, whilevarying the number of sender-side rate-limiters. (Only oneTCP stream was active at any time.)

Fig. 10 shows mean results for at least 11 trials. Eachcurve corresponds to a rate limit; we marked cases where theachieved bandwidth was under 90% of the target limit. Onthis hardware, we could run many low-bandwidth limiterswithout much effect on CPU performance. When we usedmore than about 128 high-bandwidth limiters, both CPUutilization and TCP throughput suffered, but it is unlikelythat a real system would allow both a 10Gbps flow and lotsof smaller flows on the same server. We conclude that pipe-model rate limiting does indeed scale; [19] also supports thisclaim.

9. CONCLUSION

This report described the rationale for predictive guaran-tees in cloud networks, and the design of Cicada, a systemthat provides this abstraction to tenants. Cicada is able toprovide fine-grained, temporally- and spatially-varying guar-antees without requiring the clients to specify their demandsexplicitly. We presented a prediction algorithm using the“best expert” model, where the prediction for a future epochis a weighted linear combination of past observations, withthe weights updated and learned online in an automatic way.Using traces from HP Cloud Services, we showed that Cicadaaccurately predicts tenant bandwidth needs. We also showedthat the fine-grained structure of predictive guarantees can beused by a cloud provider to improve network utilization incertain datacenter topologies.

ACKNOWLEDGMENTS

We are indebted to a variety of people at HP Labs and HPCloud for help with data collection and for discussions of thiswork. In particular, we thank Sujata Banerjee, Ken Burden,Phil Day, Eric Hagedorn, Richard Kaufmann, Jack McCann,Gavin Pratt, Henrique Rodrigues, and Renato Santos. We alsothank John Dickerson for his help in refining the predictionalgorithm in §5.1. This work was supported in part by theNational Science Foundation under grant IIS-1065219 as wellas an NSF Graduate Research Fellowship.

11

APPENDIX: VOC PARAMETER-SELECTION

The authors of [1] note that VOC can easily be extended tohandle groups of multiple sizes (Section 3.2 of [1]), but giveno method for automatically determining what the groupsshould be, nor for choosing the best values of B and O.

To address this issue, we developed a heuristic for deter-mining the VOC groups as well as B and O. The heuristicallows groups of multiple sizes, and relies on two assump-tions. First, that connections through the aggregate layer arephysically oversubscribed by some factor Op > 1. Second,that any guarantee should be sufficient to meet the tenant’soffered network load. The alternative would be to carefullyunder-provision certain virtual network paths, which wouldincrease a job’s completion time but might also reduce thetotal cost for running some jobs. Such cost-reduction wouldrequire in-depth knowledge about the tenant’s applicationand utility functions, and is beyond the scope of this work.

Our heuristic works by starting with an initial configurationof groups, with each VM is in its own group. The heuristicmerges the groups that have the highest bandwidth betweenthem for a new configuration, and iterates on this configura-tion in the same manner, until the new configuration wastesmore bandwidth than the previous configuration.

Finding B and O for a particular configuration is easy, sinceCicada collects measurement data. Our heuristic selects theB and O that minimize wasted bandwidth while never under-allocating a path. B, then, is the maximum of any hose withinone group, and O is the maximum of any hose across groups,given the previous historical data.

Algorithm 2 VOC parameter selection.1: groupsp = list of groups, initialized to one group per VM2: Gv = groupsp // last valid configuration3: while True do4: G′ = mergeLargestClusters(groupsp)5: if G′ = groupsp then6: break7: B, O, w = getParamsAndWastedBandwidth(G)8: if O < 1 // invalid configuration then9: groupsp = G′

10: continue11: Bv, Ov, wv = getParamsAndWastedBandwidth(Gv)12: if w > wv then13: break14: else15: groupsp = Gv = G′16: B, O = findParameters(Gv)Output: B, O, Gv

10. REFERENCES

[1] BALLANI, H., COSTA, P., KARAGIANNIS, T., ANDROWSTRON, A. Towards Predictable DatacenterNetworks. In SIGCOMM (2011).

[2] BENSON, T., AKELLA, A., AND MALTZ, D. A.Network Traffic Characteristics of Data Centers in theWild. In IMC (2010).

[3] BENSON, T., ANAND, A., AKELLA, A., AND ZHANG,M. Understanding Data Center Traffic Characteristics.In WREN (2009).

[4] BODÍK, P., MENACHE, I., CHOWDHURY, M., MANI,P., MALTZ, D. A., AND STOICA, I. Surviving Failuresin Bandwidth-Constrained Datacenters. In SIGCOMM(2012).

[5] DENG, S., AND BALAKRISHNAN, H. Traffic-awareTechniques to Reduce 3G/LTE Wireless EnergyConsumption. In CoNEXT (2012).

[6] DUFFIELD, N. G., GOYAL, P., GREENBERG, A.,MISHRA, P., RAMAKRISHNAN, K. K., AND VAN DERMERWE, J. E. A Flexible Model for ResourceManagement in Virtual Private Networks. InSIGCOMM (1999).

[7] ERICKSON, D., HELLER, B., YANG, S., CHU, J.,ELLITHORPE, J., WHYTE, S., STUART, S.,MCKEOWN, N., PARULKAR, G., AND ROSENBLUM,M. Optimizing a Virtualized Data Center. InSIGCOMM Demos (2011).

[8] GREENBERG, A., HAMILTON, J. R., JAIN, N.,KANDULA, S., KIM, C., LAHIRI, P., MALTZ, D. A.,PATEL, P., AND SENGUPTA, S. VL2: A Scalable andFlexible Data Center Network. In SIGCOMM (2009).

[9] GUI, C., LU, G., WANG, H. J., YANG, S., KONG, C.,SUN, P., WU, W., AND ZHANG, Y. SecondNet: AData Center Network Virtualization Architecture withBandwidth Guarantees. In CoNEXT (2010).

[10] HAJJAT, M., SUN, X., SUNG, Y.-W. E., MALTZ, D.,RAO, S., SRIPANIDKULCHAI, K., ANDTAWARMALANI, M. Cloudward Bound: Planning forBeneficial Migration of Enterprise Applications to theCloud. In SIGCOMM (2010).

[11] HERBSTER, M., AND WARMUTH, M. K. Tracking theBest Expert. Machine Learning 32 (1998), 151–178.

[12] JAMIN, S., SHENKER, S. J., AND DANZIG, P. B.Comparison of Measurement-based Admission ControlAlgorithms for Controlled-load Service. In Infocom(1997).

[13] KANDULA, S., SENGUPTA, S., GREENBERG, A.,PATEL, P., AND CHAIKEN, R. The Nature ofDatacenter Traffic: Measurements & Analysis. In IMC(2009).

[14] KIM, G., PARK, H., YU, J., AND LEE, W. VirtualMachines Placement for Network Isolation in Clouds.In RACS (2012).

[15] KNIGHTLY, E. W., AND SHROFF, N. B. AdmissionControl for Statistical QoS: Theory and Practice. IEEENetwork 13, 2 (1999), 20–29.

[16] LAM, T. V., RADHAKRISHNAN, S., PAN, R.,VAHDAT, A., AND VARGHESE, G. NetShare andStochastic NetShare: Predictable Bandwidth Allocation

12

for Data Centers. SIGCOMM CCR 42, 3 (2012), 5–11.[17] MALALUR, P. Traffic Delay Prediction from Historical

Data. Master’s thesis, Massachusetts Institute ofTechnology, 2010.

[18] MONTELEONI, C., BALAKRISHNAN, H., FEAMSTER,N., AND JAAKKOLA, T. Managing the 802.11Energy/Performance Tradeoff with Machine Learning.Tech. Rep. MIT-LCS-TR-971, Massachusetts Instituteof Technology, 2004.

[19] MYSORE, R. N., PORTER, G., AND VAHDAT, A.FasTrack: Enabling Express Lanes in Multi-TenantData Center. In CoNEXT (2013).

[20] NIU, D., FENG, C., AND LI, B. Pricing CloudBandwidth Reservations Under Demand Uncertainty.In Sigmetrics (2012).

[21] OpenStack. http://openstack.org.[22] POPA, L., KUMAR, G., CHOWDHURY, M.,

KRISHNAMURTHY, A., RATNASAMY, S., ANDSTOICA, I. FairCloud: Sharing the Network in CloudComputing. In SIGCOMM (2012).

[23] RAGHAVAN, B., VISHWANATH, K., RAMABHADRAN,S., YOCUM, K., AND SNOEREN, A. C. Cloud Controlwith Distributed Rate Limiting. In SIGCOMM (2007).

[24] RASMUSSEN, A., CONLEY, M., KAPOOR, R., LAM,V. T., PORTER, G., AND VAHDAT, A. Themis: AnI/O-Efficient MapReduce. In SOCC (2012).

[25] RASMUSSEN, A., PORTER, G., CONLEY, M.,MADHYASTHA, H., MYSORE, R. N., PUCHER, A.,AND VAHDAT, A. TritonSort: A Balanced Large-ScaleSorting System. In NSDI (2011).

[26] RODRIGUES, H., SANTOS, J. R., TURNER, Y.,SOARES, P., AND GUEDES, D. Gatekeeper:Supporting Bandwidth Guarantees for Multi-tenantDatacenter Networks. In WIOV (2011).

[27] ROUGHAN, M., THORUP, M., AND ZHANG, Y. TrafficEngineering with Estimated Traffic Matrices. In IMC(2003).

[28] sFlow.http://sflow.org/about/index.php.

[29] WANG, H., XIE, H., QIU, L., YANG, Y. R., ZHANG,Y., AND GREENBERG, A. COPE: Traffic Engineeringin Dynamic Networks. In SIGCOMM (2006).

[30] XIE, D., DING, N., HU, Y. C., AND KOMPELLA, R.The Only Constant is Change: IncorporatingTime-Varying Network Reservations in Data Centers.In SIGCOMM (2012).

[31] ZHANG, Y., ROUGHAN, M., DUFFIELD, N., ANDGREENBERG, A. Fast Accurate Computation ofLarge-Scale IP Traffic Matrices from Link Loads. InSigmetrics (2003).

13