Yank: Enabling Green Data Centers to Pull the PlugRahul Singh, David Irwin, Prashant Shenoy, and K.K. Ramakrishnan‡

University of Massachusetts Amherst ‡AT&T Labs - Research

Abstract

Balancing a data center’s reliability, cost, and carbonemissions is challenging. For instance, data centers de-signed for high availability require a continuous flowof power to keep servers powered on, and must limittheir use of clean, but intermittent, renewable energysources. In this paper, we present Yank, which usesa transient server abstraction to maintain server avail-ability, while allowing data centers to “pull the plug” ifpower becomes unavailable. A transient server’s defin-ing characteristic is that it may terminate anytime after abrief advance warning period. Yank exploits the advancewarning—on the order of a few seconds—to providehigh availability cheaply and efficiently at large scales byenabling each backup server to maintain “live” memoryand disk snapshots for many transient VMs. We imple-ment Yank inside of Xen. Our experiments show thata backup server can concurrently support up to 15 tran-sient VMs with minimal performance degradation withadvance warnings as small as 10 seconds, even whenVMs run memory-intensive interactive web applications.

1 IntroductionDespite continuing improvements in energy efficiency,data centers’ demand for power continues to rise, in-creasing by an estimated 56% from 2005-2010 and ac-counting for 1.7-2.2% of electricity usage in the UnitedStates [16]. The rise in power usage has led data cen-ters to experiment with the design of their power deliv-ery infrastructure, including the use of renewable energysources [3, 20, 21]. For instance, Apple’s goal is to runits data centers off 100% renewable power; its newestdata center includes a 40MW co-located solar farm [3].

Thus, determining the characteristics of the powerinfrastructure—its reliability, cost, and carbonemissions—has now become a key element of datacenter design [5]. Balancing these characteristics ischallenging, since providing a reliable supply of poweris often antithetical to minimizing costs (capital or oper-ational) and emissions. A state-of-the-art power deliveryinfrastructure designed to ensure an uninterrupted flowof high quality power is expensive, possibly including i)connections to multiple power grids, ii) on-site backupgenerators, and iii) an array of universal power supplies(UPSs) that both condition grid power and guaranteeenough time after an outage to spin-up and transition

power to generators. Unfortunately, while renewableenergy has no emissions, it is unreliable—generatingpower only intermittently based on uncontrollable envi-ronmental conditions—which limits its broad adoptionin data centers designed for high reliability.

Prior research focuses on balancing a data center’sreliability, cost, and carbon footprint by optimizing thepower delivery infrastructure itself, while continuing toprovide a highly reliable supply of power, e.g., using en-ergy storage technologies [13, 14, 32] or designing flex-ible power switching or routing techniques [10, 24]. Incontrast, we target a promising alternative approach: re-laxing the requirement for a continuous power sourceand then designing high availability techniques to ensuresoftware services remain available during unexpectedpower outages or shortages. We envision data centerswith a heterogeneous power delivery infrastructure thatincludes a mix of servers connected to power supplieswith different levels of reliability and cost. While someservers may continue to use a highly reliable, but expen-sive, infrastructure that includes connections to multipleelectric grids, high-capacity UPSs, and backup genera-tors, others may connect to only a single grid and usecheaper lower-capacity UPSs, and still others may relysolely on renewables with little or no UPS energy buffer.As we discuss in Section 2, permitting even slight reduc-tions in the reliability of the power supply has the poten-tial to decrease a data center’s cost and carbon footprint.

To maintain server availability while also allowingdata centers to “pull the plug” on servers if the level ofavailable power suddenly changes, i.e., is no longer suf-ficient to power the active set of of servers, we introducethe abstraction of a transient server. A transient server’sdefining characteristic is that it may terminate anytimeafter an advance warning period. Transient servers arisein many scenarios. For example, spot instances in Ama-zon’s Elastic Compute Cloud (EC2) terminate after abrief warning if the spot price ever exceeds the instance’sbid price. As another example, UPSs built into racks pro-vide some time after a power outage before servers com-pletely lose power. In this paper, we apply the abstractionto green data centers that use renewables to power a frac-tion of servers, where the length of the warning periodis a function of UPS capacity or expected future energyavailability from renewables. In this context, we showthat transient servers are cheaper and greener to oper-ate than stable servers, which assume continuous power,

since they i) do not require an expensive power infras-tructure that ensures 24x7 power availability and ii) candirectly use intermittent renewable energy.

Unfortunately, transient servers expose applications tovolatile changes in their server allotment, which degradeperformance. Ideally, applications would always use asmany transient servers as possible, but seamlessly tran-sition to stable servers whenever transient servers be-come unavailable. To achieve this ideal, we proposesystem support for transient servers, called Yank, thatmaintains “live” backups of transient virtual machines’(VMs’) memory and disk state on one or more stablebackup servers. Yank extends the concept of whole sys-tem replication popularized by Remus [7] to exploit anadvance warning period, enabling each backup server tosupport a many transient VMs. Highly multiplexing eachbackup server is critical to preserving transient servers’monetary and environmental benefits, allowing them toscale independently of the number of stable servers.

Importantly, the advance warning period eliminatesthe requirement that transient VMs always maintain ex-ternal synchrony [23] with their backup to ensure cor-rect operation, opening up the opportunity for both i) alooser form of synchrony and ii) multiple optimizationsthat increase performance and scalability. Our hypoth-esis is that a brief advance warning—on the order of afew seconds—enables Yank to provide high availabilityin the face of sudden changes in available power, cheaplyand efficiently at large scales. In evaluating our hypoth-esis, we make the following contributions.Transient Server Abstraction. We introduce the tran-sient server abstraction, which supports a relaxed variantof external synchrony. Our variant, called just-in-timesynchrony, exploits an advance warning that only ensuresconsistency with its backup before termination. We showhow just-in-time synchrony applies to advance warningswith different durations, e.g., based on UPS capacity, anddynamics, e.g., based on intermittent renewables.Performance Optimizations. We present multiple op-timizations that further exploit the advance warning toscale the number of transient VMs each backup serversupports without i) degrading VM performance duringnormal operation, ii) causing long downtimes when tran-sient servers terminate, and iii) consuming excessive net-work resources. Our optimizations leverage basic in-sights about memory usage to minimize the in-memorystate each backup server must write to stable storage.Implementation and Evaluation. We implement Yankinside the Xen hypervisor and evaluate its performanceand scalability in a range of scenarios, including withdifferent size UPSs and using renewable energy sources.Our experiments demonstrate that a backup server canconcurrently support up to 15 transient VMs with min-imal performance degradation using an advance warn-

Technique Overhead Extra Cost WarningLive Migration None None Lengthy

Yank Low Low ModestHigh Availability High High None

Table 1: Yank has less overhead and cost than high avail-ability, but requires less warning than live migration.

ing as small as 10 seconds, even when running memory-intensive interactive web applications, which is a chal-lenging application for whole system replication.

2 Motivation and BackgroundWe first define the transient server abstraction before dis-cussing Yank’s use of the abstraction in green data cen-ters that use intermittent renewable power sources.Transient Server Abstraction. We assume a virtual-ized data center where applications run inside VMs onone of two types of physical servers: (i) always-on stableservers with a highly reliable power source and (ii) tran-sient servers that may terminate anytime. Central to ourwork is the notion of an advance warning, which signalsthat a transient server will shutdown after a delay Twarn.

Once a transient server receives an advance warning,a data center must move any VMs (and their associatedstate) hosted on the transient server to a stable server tomaintain their availability. Depending on Twarn’s dura-tion, two solutions exist to transition a VM to a stableserver. If Twarn is large, it may be possible to live mi-grate a VM from a transient to a stable server. VM migra-tion requires copying the memory and disk (if necessary)state [6], so the approach is only feasible if Twarn is longenough to accommodate the transfer. Completion timesfor migration are dependent on a VM’s memory and disksize, with prior work reporting times up to one minute forVMs with only 1GB memory and no disk state [1, 19].

An alternative approach is to employ a high availabil-ity mechanism, such as Remus [7, 22], which requiresmaintaining a live backup copy of each transient VM ona stable server. In this case, a VM transparently failsover to the stable server whenever its transient serverterminates. While the approach supports warning timesof zero, it requires the high runtime overhead of con-tinuously propagating state changes from the VM to itsbackup. In some cases, memory-intensive, interactiveworkloads may experience 5X degradation in latency [7].Supporting an advance warning of zero also imposes ahigh cost, requiring a backup server to keep VM mem-ory snapshots resident in its own memory. In essence,supporting a warning time of zero requires a 1:1 ratio be-tween transient and backup servers. Unfortunately, stor-ing memory snapshots on disk is not an option, sinceit would further degrade performance by reducing thememory bandwidth (∼3000MB/s) of primary VMs to thedisk bandwidth (<100MB/s) of the backup server.

UPS (N+1)UPS (Smaller Capacity)ATS ATS

Solar Wind

On-Site Renewables Grid Energy

PDU PDU PDU PDU

RACKRACK

RACKTransient Servers Stable Servers

Figure 1: A data center with transient servers poweredby renewables and low-capacity UPSs, and stable serverspowered by the grid and redundant, high-capacity UPSs.

Live migration and high availability represent two ex-treme points in the design space for handling transientservers. The former has low overhead but requires longwarning times, while the latter has high overhead buthandles warning times of zero. Yank’s goal is to exploitthe middle ground between these two extremes, as out-lined in Table 1, when a short advance warning is insuffi-cient to complete a live migration, but does not necessar-ily warrant the overhead of externally synchronous livebackups of VM memory and disk state. Yank optimizesfor modest advance warnings by maintaining a backupcopy, similar to Remus, of each transient VM’s mem-ory and disk state on a stable backup server. However,Yank focuses on keeping costs low, by highly multiplex-ing each backup server across many transient VMs.

As we discuss, our approach requires storing portionsof each VM’s memory backup on stable storage. Weshow that for advance warnings of a few seconds, Yankprovides similar failover properties as high availability,but with an overhead and cost closer to live migration.In fact, Yank devolves to high availability for a warningtime of zero, and reduces to a simple live migration asthe warning time becomes larger. Yank focuses on sce-narios where there is an advance warning of a fail-stopfailure. Many of these failures stem from power outageswhere energy storage provides the requisite warning.Green Data Center Model. Figure 1 depicts the datacenter infrastructure we assume in our work. As shown,the data center powers servers using two sources: (i) on-site renewables, such as solar and wind energy, and (ii)the electric grid. Recent work proposes a similar archi-tecture for integrating renewables into data centers [18].

We assume that the on-site renewable sources powera significant fraction of the data center’s servers. How-ever, since renewable generation varies based on envi-ronmental conditions, this fraction also varies over time.While UPSs are able to absorb short-term fluctuations inrenewable generation, e.g, over time-scales of seconds tominutes caused by a passing cloud or a temporary dropin wind speed, long-term fluctuations require switchingservers to grid power or temporarily deactivating them.

A key assumption in our work is that it is feasible toswitch some, but not all, servers from renewable sourcesto the grid to account for these power shortfalls.

The constraint of powering some, but not all, serversfrom the grid arises if a data center limits its peak powerusage to reduce its electricity bill. Since peak power dis-proportionately affects the electric grid’s capital and op-erational costs, utilities routinely impose a surcharge onlarge industrial power consumers based solely on theirpeak demand, e.g., the maximum average power con-sumed over a 30 minute rolling window [10, 13]. Thus,data centers can reduce their electricity bills by cappinggrid power draw. In addition, to scale renewable deploy-ments, data centers will increasingly need to handle theirpower variations locally, e.g., by activating and deactivat-ing servers, since i) relying on the grid to absorb varia-tions is challenging if renewables contribute a large frac-tion (∼20%) of grid power and ii) absorbing variationsentirely using UPS energy storage is expensive [13]. Inthe former case, rapid variations from renewables couldcause grid instability, since generators may not be agileenough to balance electricity’s supply and demand.

Thus, in our model, green data centers employ bothstable and transient servers. Both server types requireUPSs to handle short-term power fluctuations. How-ever, we expect transient servers to require only a fewseconds of expensive UPS capacity to absorb short-termpower fluctuations, while stable servers may require tensof minutes of capacity to permit time to spin-up and tran-sition to backup generators in case of a power outage.

3 Yank DesignSince Yank targets modest-sized advance warnings onthe order of a few seconds, it cannot simply migrate atransient VM to a stable server after receiving a warn-ing. To support shorter warning times, one option is tomaintain backup copies (or snapshots) of a VM’s mem-ory and disk state on a dedicated stable server, and thencontinuously update the copies as they change. In thiscase, if a transient VM terminates, this backup server canrestart the VM using the latest memory and disk snap-shot. A high availability technique must commit changesto a VM’s memory and disk state to a backup server fre-quently enough to support a warning time of zero. How-ever, supporting a warning time of zero necessitates a 1:1ratio between transient and backup servers, eliminatingtransient servers’ monetary and environmental benefits.

In contrast, Yank leverages the advance warning timeto scale the number of transient servers independently ofthe number of backup servers by controlling when andhow frequently transient VMs commit state updates tothe backup server. In essence, the warning time Twarn

limits the amount of data a VM can commit to its backupserver after receiving a warning. Thus, during normal op-

T1 T2T1

Transient Server

Transient VMs

T3 T4

Transient Server

Transient VMs

T5

Tn-1 Tn

Transient Server

Transient VMs

T1 T2

Stable Server

Tn-1 Tn

Stable Server

1

1

1

3

3

3

4

4

5

5

In-Memory Queues

MemorySnapshots

Tn

Backup Server (Memory)

BackupEngine

T1 Tn

In-Memory Queues

DiskSnapshots

Tn

Backup Server (Disk)

BackupEngine

1

T1 Tn

T1

T11

2

2

Snapshot Manager

Snapshot Manager

Snapshot ManagerRestoration

Service

Restoration Service

Figure 2: Yank’s Design and Basic Operation

eration, a transient VM need only ensure the size of dirtymemory pages and disk blocks remains below this limit.Maintaining this invariant guarantees that no update willbe lost if a VM terminates after a warning, while pro-viding additional flexibility over when to commit state.To keep its overhead and cost low, Yank highly multi-plexes backup servers, allowing each to support many(>10) transient VMs by i) storing VM memory and disksnapshots, in part, on stable storage and ii) using mul-tiple optimizations to prevent saturating disk bandwidth.Thus, given an advance warning, Yank supports the samefailure properties as high availability, but uses fewer re-sources, e.g., hardware or power, for backup servers.

3.1 Yank ArchitectureFigure 2 depicts Yank’s architecture, which includes asnapshot manager on each transient server, a backupengine on each stable backup server, and a restora-tion service on each stable (non-backup) server. Wefocus primarily on how Yank maintains memory snap-shots at the backup server, since we assume each backupserver cannot keep memory snapshots for all transientVMs resident in its own memory. Thus, Yank mustmask the order-of-magnitude difference between a tran-sient VM’s memory bandwidth (∼3000MB/s) and thebackup server’s disk bandwidth (< 100MB/s). By con-trast, maintaining disk snapshots poses less of a per-formance concern, since the speed of a transient VM’sdisk and its backup server’s disk are similar in magni-tude. This characteristic combined with a multi-secondwarning time permits asynchronous disk mirroring to abackup server without significant performance degrada-tion. Thus, while many of our optimizations below applydirectly to disk snapshots, Yank currently uses off-the-shelf software (DRBD [9]) for disk mirroring.

Figure 2 also details Yank’s functions. The snapshotmanager executes within the hypervisor of each transientserver and tracks the dirty memory pages of its resi-dent VMs, periodically transmitting these pages to thebackup engine, running at the backup server (1). The

backup engine then queues each VM’s dirty memorypages in its own memory before writing them to disk(2). Yank includes a service that monitors UPS state-of-charge, via the voltage level across its diodes, andtranslates the readings into a warning time based on thepower consumption of transient servers (3). The serviceboth i) informs backup and transient servers when thewarning time changes and ii) issues warnings to transientand backup servers of an impending termination due toa power shortage. Since Yank depends on warning timeestimates, the service above runs on a stable server. Wediscuss warning time estimation further in Section 3.5.

Upon receiving a warning (3), the snapshot managerpauses its VMs and commits any dirty pages to thebackup engine before the transient server terminates.The backup engine then has two options, assuming it istoo resource-constrained to run VMs itself: either storethe VMs’ memory images on disk for later use, or mi-grate the VMs to another stable (non-backup) server (4).Yank executes a simple restoration service on each stable(non-backup) server to facilitate rapid VM migration andrestoration after a transient server terminates.

3.2 Just-in-Time SynchronySince Yank receives an advance warning of time Twarn

before a transient server terminates, its VM memorysnapshots stored on the backup server need not maintainstrict external synchrony [23]. Instead, upon receivinga warning of impending termination, Yank only has toensure what we call just-in-time synchrony: a transientVM and its memory snapshot on the backup server arealways capable of being brought to a consistent state be-fore termination. To guarantee just-in-time synchrony,as with external synchrony, the snapshot manager tracksdirty memory pages and transmits them to the backup en-gine, which then acknowledges their receipt. However,unlike external synchrony, just-in-time synchrony onlyrequires the snapshot manager to buffer a VM’s exter-nally visible, e.g., network or disk, output when the sizeof the dirty memory pages exceed an upper threshold Ut,such that it is impossible to commit any more dirty pagesto the backup engine within time Twarn.

In the worst case, with a VM that dirties pages fasterthan the backup engine is able to commit them, the snap-shot manager is continually at the threshold, and Yankreverts to high availability-like behavior by always de-laying the VM’s externally visible output until its mem-ory snapshot is consistent. Since we assume the backupserver is not able to keep every transient VM memorysnapshot resident in its own memory, the speed of thebackup engine’s disk limits the rate it is able to commitpage changes. While memory bandwidth is an order ofmagnitude (or more) greater than disk (or network) band-width, Yank benefits from well-known characteristics of

0 50

100 150 200 250 300 350

0 5 10 15 20 25 30Tota

l D

irty

Pages (

MB

)

Time (mins)

SpecJBBTPCW

Kernel Compile

Figure 3: Working set size over time for three bench-marks: SPECjbb, TPCW, and Linux kernel compile.

typical in-memory application working sets to preventsaturating disk (or network) bandwidth. Specifically, thesize of in-memory working sets tend to i) grow slowlyover time and ii) be smaller than the total memory [8].

Slow growth stems from applications frequently re-writing the same memory pages, rather than always writ-ing new ones. Yank only has to commit the last re-writeof a dirty page (and not the intervening writes) to thebackup server after reaching its upper threshold of dirtypages Ut. In contrast, to support termination with no ad-vance warning, a VM must commit nearly every memorywrite to the backup server. In addition, small workingsets enable the backup engine to keep most VMs’ work-ing sets in memory, reducing the likelihood of saturatingdisk bandwidth from writing dirty memory pages to disk.Recent work extends this insight to collections of VMs indata centers, showing that while the size of a single VM’sworking set may experience temporary bursts in memoryusage, the bursts are often brief and not synchronizedacross VMs [33]. Yank relies on these observations inpractice to highly multiplex each backup server’s mem-ory without saturating disk bandwidth or degrading tran-sient VM performance during normal operation.

To confirm the characteristics above, we conducteda simple experiment: Figure 3 plots the dirty memorypages measured every 100ms for a VM with 1GB mem-ory over a thirty minute period with three different ap-plications: 1) the SPECjbb benchmark with 400 ware-houses, 2) the TPC-W benchmark with 100 clients per-forming a browsing workload and 3) a Linux kernel (v3.4) compile. The experiment verifies the observationsabove, namely that in each case i) the VM dirties lessthan 350MB or 35% of the available memory and ii) afterexperiencing an initial burst in memory usage the work-ing set grows slowly over time.

Yank also relies on the observations above in setting itsupper threshold Ut. To preserve just-in-time synchrony,this threshold represents the maximum size of the dirtypages the snapshot manager is capable of committing tothe backup engine within the warning time. Our premiseis that as long as the backup engine is able to keep eachVM’s working set in memory, even if all VMs simulta-

neously terminate, it should be able to commit any out-standing dirty pages without saturating disk bandwidth.Thus, Ut is a function of the warning time, the avail-able network bandwidth between transient and backupservers, and the number of transient servers that may si-multaneously terminate. For instance, for a single VMhosted on a transient server using a 1Gbps network link,a one second advance warning results in Ut=125MB. InFigure 3 above, Ut=125MB would only force SpecJBBto pause briefly on startup (where its usage briefly burstsabove 125MB/s). The other applications never allocatemore than 125MB in one second.

Of course, to support multiple VMs terminating simul-taneously requires a lower Ut. However, as discussed inSection 3.5, Yank bases its warning time estimates onan “empty” UPS being at 40-50% depth-of-discharge.Thus, Ut need not be exactly precise, providing timeto handle unlikely events, such as a warning coincidingwith a correlated burst in VM memory usage, which mayslow down commits by saturating the backup engine’sdisk bandwidth, or all VMs simultaneously terminating.

3.3 Optimizing VM PerformanceFor correctness, just-in-time synchrony only requirespausing a transient VM and committing dirty pages to thebackup engine once their size reaches the upper thresh-old Ut, described above. A naıve approach only employsa single threshold at Ut by simply committing all dirtypages once their size reaches Ut. However, this approachhas two drawbacks. First, it forces the VM to inevitablydelay the release of externally visible output each timeit reaches Ut, effectively pausing the VM from the per-spective of external clients. If the warning time is long,e.g., 5-10 seconds, then the Ut will be large, causing longpauses. Second, it causes bursts in network traffic that af-fect other network applications.

To address these issues, the snapshot manager alsouses a lower threshold Lt. Once the size of the dirtypages reaches Lt, it begins asynchronously committingdirty pages to the backup engine until the size is lessthan Lt. Algorithm 1 shows pseudo-code for the snap-shot manager. The downside to using Lt is that the snap-shot manager may end up committing more pages thanwith a single threshold, since it may unnecessarily com-mit the same memory page multiple times. Yank usestwo techniques to mitigate this problem. First, to deter-mine the order to commit pages, the snapshot managerassociates a timestamp with each dirty page and uses aLeast Recently Used (LRU) policy to prevent commit-ting pages that are being frequently re-written. Second,the snapshot manager adaptively sets the lower thresholdto be just higher than the VM’s working set, since theworking set contains the pages a VM is actively writing.

As we discuss in Section 5, our results show that as

Algorithm 1: Snapshot Manager’s Algorithm forCommitting Dirty Pages to the Backup Engine

1 Initialize Dirty Page Bitmap;2 while No Warning Signal do3 Check Warning Time Estimate;4 Convert Warning Time to Upper Threshold (Ut);5 Get Num. Dirty Pages;6 if Num. Dirty Pages > Ut then7 Buffer Network Output of Transient VM;8 Transmit Dirty Pages to Backup Engine to9 Reduce Num. Dirty Pages to Lower Threshold (Lt);

10 Wait for Ack. from Backup Engine;11 Unset Dirty Bitmap for Pages Sent;12 Release Buffered Network Packets;13 end14 if Num. Dirty Pages > Lt then15 Transmit Dirty Pages to Backup Engine at Specified Rate;16 Wait for Ack. from Backup Engine;17 Unset Dirty Bitmap for Pages Sent;18 end19 end20 Warning Received;21 Pause the Transient VM;22 Transmit Dirty Pages to Receiver Service;23 Destroy the Transient VM;

long as the size of the VM’s working set is less than Ut,using Lt results in smoother network traffic and fewer,shorter VM pauses. Of course, a combination of a largeworking set and short warning time may force Yank todegrade performance by continuously pausing the VMto commit frequently changing memory pages. In Sec-tion 5, we evaluate transient VM performance for a vari-ety of applications with advance warnings in the rangeof 5-10 seconds. Finally, the snapshot manager im-plements standard optimizations to reduce network traf-fic, including content-based redundancy elimination andpage deltas [34, 35]. The former technique associatesmemory pages with a hash based on their content, allow-ing the snapshot manager to send a 32b hash index ratherthan a 4kB memory page if the page is already presenton the backup server. The technique is most useful in re-ducing the overhead of committing zero pages. The lat-ter technique allows the snapshot manager to only send apage delta if it has previously committed a memory page.

3.4 Multiplexing the Backup ServerTo multiplex many transient VMs, the backup enginemust balance two competing objectives: using disk band-width efficiently during normal operation, while mini-mizing transient VM downtime in the event of a warning.

3.4.1 Maximizing Disk EfficiencyThe backup engine maintains an in-memory queue foreach transient VM to store newly committed (and ac-knowledged) memory pages. Since the backup server’smemory is not large enough to store a complete mem-ory snapshot for every transient VM it supports, it mustinevitably write pages to disk. Yank includes multipleoptimizations to prevent unnecessary disk writes.

First, when a transient VM commits a change to amemory page already present in its queue, the receiverdeletes the out-of-date memory page without writing itto disk. As a result, the backup engine can often elimi-nate disk writes for frequently changing pages, even ifthe snapshot manager commits them. Second, to fur-ther prevent unnecessary writes, the backup engine or-ders each VM’s queue using an LRU policy. Our useof LRU in both the snapshot manager (when determin-ing which pages to commit on the transient VM) and thebackup engine (when determining which pages to writeto disk on the backup server) follows the same princi-ples as a standard hierarchical cache. In addition, to ex-ploit the observation that bursts in VM memory usage arenot highly correlated, the backup engine selects pages towrite to disk by applying LRU globally across all VMqueues. Since it allocates a fixed amount of memory forall queues (near the backup server’s total memory), theglobal LRU policy allows each VM’s queue to grow andshrink as its working set size changes.

Finally, to further maximize disk efficiency, thebackup engine could also use a log-structured file sys-tem [25], since its workload is write mostly, read rarely(only in the event of a failure). We discuss this designalternative and its implications in the next section.

3.4.2 Minimizing DowntimeThe primary bottleneck in restoring a transient VM af-ter a failure is the time required to read its memory statefrom the backup server’s disk. Thus, to minimize down-time, as soon as a transient VM receives a warning, thebackup engine immediately halts writing dirty pages todisk and begins reading the VM’s existing (out-of-date)memory snapshot from disk, without synchronizing itwith the queue of dirty page updates. Instead, the backupengine first sends the VM memory snapshot from disk toa restoration service, running on the destination stable(non-backup) server. We assume that an exogenous pol-icy exists to select a destination stable server in advanceto run a transient VM if its server terminates. In parallel,the backup engine also sends the VM’s in-memory queueto the restoration service, after updating it with any out-standing dirty pages from the halted transient VM. To re-store the VM, the restoration service applies the updatesfrom the in-memory queue to the memory snapshot fromthe backup server’s disk without writing to its own disk.Importantly, the sequence only requires reading a VM’smemory snapshot from disk once, which begins as soonas the backup engine receives the warning. Note that ifmultiple transient VMs fail simultaneously, the backupengine reads and transmits memory snapshots from diskone at a time to maximize disk efficiency and minimizedowntime across all VMs.

There are two design alternatives for determining how

the backup engine writes dirty page updates to its disk.As mentioned above, one approach is to use a log-structured file system. While a pure log-structured filesystem works well during normal operation, it results inrandom-access reads of a VM’s memory snapshot storedon disk during failover, significantly increasing down-time (by ∼100X, the difference between random-accessand sequential disk read bandwidth). In addition, main-taining log-structured files may lead to large log fileson the backup server over time. The other alternativeis to store each VM memory snapshot sequentially ondisk, which results in slow random-access writes duringnormal operation but leads to smaller downtimes duringfailover because of faster sequential reads. Of course,this alternative is clearly preferable for solid state drives,since there is no difference between sequential and ran-dom write bandwidth. Considering these tradeoffs, inYank’s current implementation we use this design alter-native to minimize downtime during failure.

3.5 Computing the Warning TimeYank’s correct operation depends on estimates of the ad-vance warning time. There are multiple ways to computethe warning time. If transient servers connect to the grid,the warning time is static and based on server power con-sumption and UPS energy storage capacity. In the eventof a power outage, if the UPS capacity is N watt-hoursand the aggregate maximum server power is M watts,then the advance warning time is N

M . Alternatively, thewarning time may vary in real-time if green data centerscharge UPSs from on-site renewables. In this case, theUPSs’ varying state-of-charge dictates the warning time,ensuring that transient servers are able to continue oper-ation for some period if renewable generation immedi-ately drops to zero. Note that warning time estimates donot need to be precisely accurate. Since, to minimizetheir amortized cost, data centers should not routinelyuse UPSs beyond a 40%-50% depth-of-discharge, evenan “empty” UPS has some remaining charge to mind-the-gap and compensate for slight inaccuracies in warn-ing time estimates. As a result, a natural buffer exists ifwarning time estimates are too short, e.g., by 1%-40%,although repeated inaccuracies degrade UPS lifetime.

4 Yank ImplementationYank’s implementation is available athttp://yank.cs.umass.edu. We implement the snap-shot manager by extending Remus inside the Xenhypervisor (v4.2). By default, Remus tracks dirty pagesover short epochs (∼100ms) using shadow page tablesand pausing VMs each epoch to copy dirty memorypages to a separate buffer for transmission to the backupserver. While VMs may speculatively execute aftercopying dirty pages to the buffer, but before receiving

Transient VM

DirtyPage

Tracker

DRBDEngine

NetworkBufferingModule

NetworkOutput

DiskWrites

MemoryWrites

Transient Server

Xen-Based Snapshot Manager

Memory Updates

DiskUpdates

Buffer n/wmessage

Backup Server

BackupEngine

Figure 4: Snapshot Manager on the Transient Server

an acknowledgement from the backup server, they mustbuffer external network or disk output to preserve exter-nal synchrony. Remus only releases externally-visibleoutput from these buffers after the backup server hasacknowledged receiving dirty pages from the last epoch.Of course, by conforming to strict external synchrony,Remus enables a higher level of protection than Yank,including unexpected failures with no advance warn-ing, e.g., fail-stop disk crashes. Although our currentimplementation only tracks dirty memory pages, it isstraightforward to extend our approach to disk blocks.

Rather than commit dirty pages to the backup serverevery epoch, our snapshot manager uses a simple bitmapto track dirty pages and determine whether to committhese pages to the backup engine based on the upperand lower threshold, Ut and Lt. In addition, rather thancommit CPU state each epoch, as in Remus, the snap-shot manager only commits CPU state when it receivesa warning that a transient server will terminate. Imple-menting the snapshot manager required adding or mod-ifying roughly 600 lines of code (LOC), primarily infiles related to VM migration, save/restore, and networkbuffering, e.g., xc domain save.c, xg save restore.h, andsch plug.c. Finally, the snapshot manager includesa simple /proc interface to receive notifications aboutwarnings or changes in the warning time. Figure 4 de-picts the snapshot manager’s implementation. As men-tioned in the previous section, our current implementa-tion uses DRBD to mirror disk state on a backup server.

Instead of modifying Xen, we implement Yank’sbackup engine “from scratch” at user-level for greaterflexibility in controlling its in-memory queues and diskwriting policy. The implementation is a combination ofPython and C, with a Python front-end (∼300LOC) thataccepts network connections and forks a backend C pro-cess (∼1500LOC) for each transient VM, as describedbelow. Since the backup engine extends Xen’s live mi-gration and Remus functionality, the front-end listens onthe same port (8002) that Xen uses for live migration.Figure 5 shows a detailed diagram of the backup engine.

For each transient VM, the backend C process accepts

Backup Engine

RECEIVER PROCESS

PRODUCERTHREAD

BUFFER

CONSUMERTHREAD

RECEIVER PROCESS

PRODUCERTHREAD

BUFFER

CONSUMERTHREADProducer

ThreadConsumerThread

DRBDEngine

Per-VM Backend Process

Backup Server

In-MemoryQueue

Figure 5: Backup Engine on the Backup Server

dirty page updates from the snapshot manager and sendsacknowledgements. Each update includes the number ofpages in the update, as well as each page’s page num-ber and contents (or a delta from the previous page sent).The process then places each update in an in-memoryproducer/consumer queue. To minimize disk writes, asdescribed in Section 3.4.1, before queuing the update,the process checks the queue to see if a page already hasa queued update. If so, the process merges the two up-dates. To perform these checks, the process maintainsa hashtable that maps page numbers to their positionin their queue. The process’s consumer thread then re-moves pages from the queue (in LRU order based on atimestamp) and writes them to disk. In the current im-plementation, the backend process stores VM memorypages sequentially in a file on disk. For simplicity, thefile’s format is the same as Xen’s format for storing savedVM memory images, e.g., via xm save. As discussedin Section 3.4.2, this results in low downtimes during mi-gration, but lower performance during normal operation.

Finally, we implement Yank’s restoration service(∼300LOC) at user-level in C. The daemon accepts aVM memory snapshot and an in-memory queue of pend-ing updates, and then applies the updates to the snap-shot without writing to disk. Since our implementationuses Xen’s image format, the service uses xm restorefrom the resulting in-memory file to re-start the VM.

5 Experimental EvaluationWe evaluate Yank’s network overhead, VM performance,downtime after a warning, and scalability, and then con-duct case studies using real renewable energy sources.While our evaluation does not capture the full range ofdynamics present in a production data center, it doesdemonstrate Yank’s flexibility to handle a variety of dy-namic and unexpected operating conditions.

5.1 Experimental SetupWe run our experiments on 20 blade servers with 2.13GHz Xeon processors with 4GB of RAM connected tothe same 1Gbps Ethernet switch. Each server runningthe snapshot manager uses our modified Xen (v4.2) hy-pervisor, while those running the backup engine and therestoration service use unmodified Xen (v4.2). In ourexperiments, each transient VM uses one CPU and 1GBRAM, and runs the same OS and kernel (Ubuntu 12.04,

Linux kernel 3.2.0). We experiment with three bench-marks from Figure 3—TPC-W, SPECjbb, and a Linuxkernel compile—to stress Yank in different ways.TPC-W is a benchmark web application that emulatesan online store akin to Amazon. We use a Java servlets-based multi-tiered configuration of TPC-W that usesApache Tomcat (v7.0.27) as a front end and MySQL(v5.0.96) as a database backend. We use additional VMsto run clients that connect to the TPC-W shopping web-site. Our experiments use 100 clients connecting to TPC-W and performing the “browsing workload” where 95%of the clients only browse the website and the remaining5% execute order transactions. TPC-W allows us to mea-sure the influence of Yank on the response time perceivedby the clients of an interactive web application.SPECjbb 2005 emulates a warehouse application forprocessing customer orders using a three-tier architec-ture comprising web, application, and database tiers. Thebenchmark predominantly exercises the middle tier thatimplements the business logic. We execute the bench-mark on a single server in standalone mode using localapplication and database servers. SPECjbb is memory-intensive, dirtying memory at a fast rate, which stressesYank’s ability to maintain snapshots on the backup serverwithout degrading VM performance.Linux Kernel Compile compiles v3.5.3 of the kernel,along with all of its modules. The kernel compilationstresses both the memory and disk subsystems and is rep-resentative of a common development workload.

Note that the first two benchmarks are web applica-tions, which are challenging due to their combination ofinteractivity and rapid writes to memory. We focus oninteractive applications rather than non-interactive batchjobs, since the latter are more tolerant to delays and per-mit simple scheduling approaches to handling periodicpower shortfalls, e.g., [2, 11, 12, 17]. Yank is applica-ble to batch jobs, although instead of scheduling them itshifts them to and from transient servers as power varies.

5.2 Benchmarking Yank’s PerformanceNetwork Overhead. Scaling Yank requires every tran-sient VM to continuously send memory updates to thebackup server. We first evaluate how much networktraffic Yank generates, and how its optimizations helpin reducing that traffic. As discussed in Section 3.3,the snapshot manager begins asynchronously commit-ting dirty pages to the backup engine after reaching alower threshold Lt. We compare this policy, which wecall async, with the naıve policy, which we call sync,that enforces just-in-time synchrony by starting to com-mit dirty pages only after their size reaches the upperthreshold Ut. Rather than commit all dirty pages whenreaching Ut, which causes long pauses, the policy com-mits pages until the dirty pages reaches 0.9 ∗ Ut. We ex-

0 2000 4000 6000 8000

10000 12000

Asyn

cA

syn

c2

Syn

cA

syn

c+

LR

UA

syn

c2

+L

RU

Syn

c+

LR

U

Asyn

cA

syn

c2

Syn

cA

syn

c+

LR

UA

syn

c2

+L

RU

Syn

c+

LR

U

Asyn

cA

syn

c2

Syn

cA

syn

c+

LR

UA

syn

c2

+L

RU

Syn

c+

LR

U

Da

ta T

x (

MB

s)

Warning Time (secs)

Pre-WarningPost-Warning

20105

(a) Data Transferred with TPC-W

0 1000 2000 3000 4000 5000 6000

Asyn

cA

syn

c2

Syn

cA

syn

c+

LR

UA

syn

c2

+L

RU

Syn

c+

LR

U

Asyn

cA

syn

c2

Syn

cA

syn

c+

LR

UA

syn

c2

+L

RU

Syn

c+

LR

U

Asyn

cA

syn

c2

Syn

cA

syn

c+

LR

UA

syn

c2

+L

RU

Syn

c+

LR

UD

ata

Tx (

MB

s)

Warning Time (secs)

Pre-WarningPost-Warning

20105

(b) Data Transferred with Kernel Compile

0 2000 4000 6000 8000

10000 12000 14000 16000 18000

Asyn

cA

syn

c2

Syn

cA

syn

c+

LR

UA

syn

c2

+L

RU

Syn

c+

LR

U

Asyn

cA

syn

c2

Syn

cA

syn

c+

LR

UA

syn

c2

+L

RU

Syn

c+

LR

U

Asyn

cA

syn

c2

Syn

cA

syn

c+

LR

UA

syn

c2

+L

RU

Syn

c+

LR

U

Da

ta T

x (

MB

s)

Warning Time (secs)

Pre-WarningPost-Warning

20105

(c) Data Transferred with SPECjbb

Figure 6: Network overhead for each benchmark over a 15 minute period.

amine two variants of the async policy: the first one setsthe lower threshold Lt to 0.5∗Ut and the second one setsit to 0.75 ∗ Ut. Our standard async and sync policies usea FIFO queue to select pages to commit to the backupengine; we label Yank’s LRU optimization separately.

In this experiment, transient VMs draw power fromthe grid, and have a static warning time dictated by theirUPS capacity. We also limit the backup engine to usingan in-memory queue of 300MB to store memory updatesfrom each 1GB transient VM. We run each experimentfor 15 minutes before simulating a power outage by issu-ing a warning to the transient and backup server. We thenmeasure the data transferred both before and after thewarning for each benchmark. Figure 6 shows the results,which demonstrate that network usage, in terms of totaldata transferred, decreases with increasing warning time.As expected, the sync policy leads to less network us-age than either async policy, since it only commits dirtypages when absolutely necessary. However, the experi-ment also shows that combining LRU with async reducesthe network usage compared to async with a FIFO pol-icy. We see that with just a 10 second warning time,Yank sends less than 100MB of data over 15 minutesfor both TPC-W and the kernel compile, largely becauseafter their initial memory burst these applications re-write the same memory pages. For the memory-intensiveSPECjbb benchmark, a 10 second warning time resultsin poor performance and excessive network usage. How-ever, a 20 second warning time reduces network usage to<100MB over the 15 minute period.Result: The sync policy has the lowest network over-head, although async with LRU also results in low net-work overhead. With these policies, a 10 second warningleads to < 100MB of data transfer over 15 minutes.Transient VM Performance. We evaluate Yank’s effecton VM performance during normal operation by usingthe same experimental setup as before except that we donot issue any warning and only evaluate pre-warning per-formance. Here, we focus on TPC-W, since it is an in-teractive application that is sensitive to VM pauses frombuffering network or disk output. We measure the av-erage response time of TPC-W clients, while varying

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

0 5 10 20Avera

ge R

esponse T

ime (

msecs)

Warning Time (secs)

SyncAsync

Async2Sync+LRU

Async+LRUAsync2+LRU

Figure 7: TPC-W response time as warning time varies.

both the warning time and the snapshot manager’s pol-icy for committing pages to the backup engine. Figure 7shows that the async policy that selects pages to com-mit using and LRU policy results in the lowest averageresponse time. The async policy reduces VM pauses, be-cause the snapshot manager begins committing pages tothe backup as soon as it hits the lower threshold Lt ratherthan waiting until reaching Ut, and forcing a large com-mit to the backup server. The experiment also demon-strates that with even a brief five second warning, the re-sponse time is <500ms using the async policy with LRU.

By contrast, with a warning time of zero the averageresponse time rises to over nine seconds. In addition, the90th percentile response time was also near 15 seconds,indicating that the average response is not the result ofa few overly bad response times. With no warning, theVM must pause and mirror every memory write to thebackup server and receive an acknowledgement beforeproceeding. Although Remus [7] does not use our spe-cific TPC-W benchmark, their results with the SPECwebbenchmark are qualitatively similar, showing 5X worselatency scores. Thus, our results confirm that even mod-est advance warning times lead to vast improvements inresponse time for interactive applications.Result: Yank imposes minimal overhead on TPC-W dur-ing normal operation. With a brief five second warningtime, the average response time of TPC-W is 10x less(<500ms) than with no warning time (>9s).VM Downtime after a Warning. The experimentsabove demonstrate that Yank imposes modest networkand VM overhead during normal operation. In this ex-periment, we issue a warning to the transient server at

0 100 200 300 400 500 600 700

TPCW

KernelCom

pile

SpecJBB

TPCW

KernelCom

pile

SpecJBB

TPCW

KernelCom

pile

SpecJBB

TPCW

KernelCom

pile

SpecJBB

Tim

e (

secs)

Backup Engine In-Memory Queue Size

Transmit To Backup EngineUpdate VM Snapshot with In-Memory Queue

Transmit Updated VM Snapshot to StableRestore

400 MB200 MB100 MB50 MB

Figure 8: Downtime with the straightforward approach.

the end of 15 minutes and measure downtime whilethe backup engine migrates the transient VM to a sta-ble server. We compare Yank’s approach (described inSection 3.4.1), which requires only a single read of theVM’s memory image from disk, with a straightforwardapproach where the backup engine applies all updates tothe memory snapshot before migrating it to the destina-tion stable server. Note that in this latter case there is noneed for Yank’s restoration service, since the backup en-gine can simply perform a Xen stop-and-copy migrationof the consistent memory snapshot at the backup server.

Figure 8 plots the transient VM’s downtime using thestraightforward approach, and Figure 9 plots it usingYank’s approach. Each graph decomposes the down-time into each stage of the migration. In Figure 8, thelargest component is the time required to create a consis-tent memory snapshot on the backup engine by updatingthe memory snapshot on disk. In addition, we run theexperiment with different sizes of the in-memory queueto show that downtime increases with queue size, sincea larger queue size requires writing more updates to diskafter a warning. While reading the VM’s memory snap-shot from disk and transmitting it to the destination stableserver still dominates downtime using Yank’s approach(Figure 9), it is less than half than with the straight-forward approach and is independent of the queue size.Note that Yank’s downtimes are in the tens of secondsand bounded by the time to read a memory snapshotfrom disk. While these downtimes are not long enoughto break TCP connections, they are much longer than themillisecond-level downtimes seen by live migration.Result: Yank minimizes transient VM downtime after awarning to a single read of its memory snapshot fromdisk, which results in a 50s downtime for a 1GB VM.Scalability. The experiments above focus on perfor-mance with a single VM. We also evaluate how manytransient VMs the backup engine is able to support con-currently, and the resulting impact on transient VM per-formance during normal operation. Again, we focus onthe TPC-W benchmark, since it is most sensitive to VMpauses. In this case, our experiments last for 30 minutesusing a warning time of 10 seconds, and scale the numberof transient VMs running TPC-W connected to the samebackup server. We measure CPU and memory usage on

0 50

100 150 200 250 300 350

TPCW

KernelCom

pile

SpecJBB

TPCW

KernelCom

pile

SpecJBB

TPCW

KernelCom

pile

SpecJBB

TPCW

KernelCom

pile

SpecJBB

Tim

e (

secs)

Backup Engine In-Memory Queue Size

Transmit to Backup EngineTransmit In-Memory Queue to Stable

Transmit VM Snapshot to StableUpdate VM Snapshot with In-Memory Queue

Restore

400 MB200 MB100 MB50 MB

Figure 9: Downtime with Yank’s optimizations.

the backup server, as well as the average response time ofthe TPC-W clients. Figure 10 shows the results, includ-ing the maximum of the average response time across alltransient VMs observed by the TPC-W clients, the CPUutilization on the backup server, and the backup engine’smemory usage as a percentage of total memory. The fig-ure demonstrates that, in this case, the backup server iscapable of supporting as many as 15 transient VMs with-out the average client response time exceeding 700ms.Note that without using Yank the average response timefor TPC-W clients running our workload is 300ms. Inaddition, even when supporting 15 VMs, the backup en-gine does not completely use its entire CPU or memory.Result: Yank is able to highly multiplex each backupserver. Our experiments indicate that with a warningtime of 10 seconds, a backup server can support at least15 transient VMs running TPC-W with little performancedegradation for even a challenging interactive workload.

5.3 Case StudiesThe previous experiments benchmark different aspects ofYank’s performance. In this section, we use case studiesto show how Yank might perform in a real data centerusing renewable energy sources. We use traces from ourown solar panel and wind turbine deployments, whichwe have used in prior work [4, 26, 27, 29]. Note that inthese experiments the warning time changes as renew-able generation fluctuates, since we assume renewablescharge a small UPS that powers the servers.

5.3.1 Adapting to Renewable GenerationFigure 11 shows the renewable power generation fromcompressing a 3-day energy harvesting trace. For theseexperiments, we assume the UPS capacity dictates amaximum warning time of 20 seconds, and that eachserver requires a maximum power of 300W. Our resultsare conservative, since we assume servers always drawtheir maximum power. In the trace, at t=60, power gen-eration falls below the 300W the server requires, causingthe battery to discharge and the warning time to decrease.At t=80, power generation rises above 300W, causing thewarning time to increase. Figure 11 shows the instanta-neous response time of a TPC-W client running on a tran-sient VM as power varies. The response time rises when

0

200

400

600

800

1000

5 10 15 0

10

20

30

40

50

60

70

80

90

100T

ime (

msecs)

Perc

enta

ge(%

)

# of Transient Instances

Response TimeCPU%

Memory%

Figure 10: Yank scalability

power generation decreases, and falls when it increases.The experiment illustrates an important property of

Yank: it is capable of translating variations in poweravailability to variations in application performance,even for interactive application running on a sin-gle server. Since servers remain far from energy-proportional, decreases in power availability require datacenters to deactivate servers. Until now, the only wayto scale application performance with power was to ap-proximate energy proportionality in large clusters by de-activating a subset of servers [30]. For interactive appli-cations not tolerant to delays, this approach is not ideal,especially if applications run on small set of servers. Ofcourse, Yank’s approach complements admission con-trol policies that may simply reject clients to decreaseload, rather than satisfying existing clients with a slightlylonger response time. In many cases, simply rejectingnew or existing clients may be undesirable.Result: Yank translates variations in power availabilityto variations in application performance for interactiveapplications running on a small number of servers.

5.3.2 End-to-End ExamplesWe use end-to-end examples to demonstrate how Yankreacts to changes in renewable power by migrating tran-sient VMs between transient and stable servers.Solar Power. We first compress solar power traces from7am to 5pm on both a sunny day and a cloudy day to a2-hour period, and then scale the power level such thatthe trace’s average power is equal to a server’s maximumpower (300W). While we compress our traces to enableexperiments to finish within reasonable times, we do notintroduce any artificial variability in the power genera-tion, since renewable generation is already highly vari-able [31]. We then emulate a solar-powered transientserver using a UPS that provides a maximum warningtime of 30 seconds, although as above when power gen-eration falls below 300W the UPS discharges and thewarning time decreases. When the warning time reacheszero, Yank issues a warning and transfers the transientVM to a stable server. Likewise, when the warningtime is non-zero continuously for a minute, Yank reacti-vates the transient server and transfers the VM back to it.Again, we run TPC-W in the VM and measure response

0

100

200

300

400

500

600

700

800

900

1000

0 30 60 90 120 150 180 0

500

1000

1500

2000

2500

3000

Pow

er

(Watts)

Response T

ime (

ms)

Time (seconds)

20s

Power GeneratedResponse Time

Warning Time

Figure 11: TPC-W response time as power varies

time at the client as power generation varies.Figure 12(a) and (b) shows that for both days Yank

adapts to power variations with negligible impact on ap-plication performance. The sunny day (a) only requiresthe transient server to deactivate once, and has negligibleimpact on response time throughout the day. The cloudyday (b) requires the transient server to deactivate justseven times throughout the day. Thus, the applicationexperiences seven brief periods of downtime, roughly 50seconds in length, over the day. However, even thoughpower is more intermittent than the sunny day, outside ofthese seven periods, the impact on response time is onlyslightly higher than during the sunny day. Note that, inthis experiment, the periods where the VM executes on astable server are brief, with Yank migrating the VM backto the transient server after a short time.Wind Power. Wind power varies significantly more thansolar power. Thus, in this experiment, we use a lessconservative approach to computing the warning time.Rather than computing the warning time based on aUPS’s remaining energy, we use a simple past-predicts-future (PPF) model (from [27]) to estimate future en-ergy harvesting. The model predicts energy harvestingover the next 10 seconds will be same the same as thelast 10 seconds. As above, we compress a wind energytrace from 7am to 5pm to two hours and scale its aver-age power generation to the server’s power. Since ourPPF predictions operate over 10 second intervals, we usea UPS capacity that provides 10 seconds of power if pre-dictions are wrong. We again measure TPC-W responsetime as it shifts between a transient and stable server.

Figure 13(a) shows the PPF model accurately predictspower over these short timescales even though powergeneration varies significantly, allowing Yank to issuewarnings with only a small amount of UPS power. Fig-ure 13(b) shows the corresponding TPC-W responsetime, which is similar to the response time in the morestable solar traces. Of course, as during the cloudy daywith solar power, when wind generation drops for a longperiod there is a brief downtime as the VM migrates fromthe transient server to a stable server.Result: Yank is flexible enough to handle different levelsof intermittency in available power resulting from varia-tions in renewable power generation.

0

100

200

300

400

500

600

700

800

900

1000

0 3600 7200 0

1000

2000

Po

we

r (W

att

s)

Re

sp

on

se

Tim

e (

ms)

Time (seconds)

Power GeneratedOn-Grid Response Time

On-Solar Response TimeBattery

DowntimePower Consumed

(a) Sunny Day

0

100

200

300

400

500

600

700

800

900

1000

0 3600 7200 0

1000

2000

Pow

er

(Watts)

Response T

ime (

ms)

Time (seconds)

(b) Cloudy Day

Figure 12: Yank using solar power on both a sunny and cloudy day.

0

1000

2000

3000

4000

5000

6000

0 3600 7200

Po

we

r (W

att

s)

Time (seconds)

Power PredictionPower Actual

Power ConsumptionWarning Signal

(a) Short-term PPF prediction model

0

500

1000

1500

2000

2500

3000

3500

4000

0 3600 7200 0

1000

2000

Pow

er

(Watts)

Response T

ime (

ms)

Time (seconds)

Power GeneratedOn-Grid Response Time

On-Wind Response TimeOn-Battery Response Time

DowntimePower Consumed

(b) Wind-powered Server

Figure 13: Yank using wind power.

6 Related Work

Prior work on supporting renewables within a data centerprimarily targets non-interactive batch jobs, since thesejobs are more tolerant to long delays when renewablepower is not available. For example, GreenSlot [11] isa general batch job scheduler that uses predictions of fu-ture energy harvesting to align job execution with pe-riods of high renewable generation. Similarly, relatedwork [2, 12] proposes similar types of scheduling algo-rithms that specifically target MapReduce jobs. Thesesolutions only support non-interactive batch jobs, and, insome cases, specifically target solar power [10], whichis more predictable than wind power. Yank takes a dif-ferent approach to support interactive applications run-ning off renewable power. However, Yank’s snapshots ofmemory and disk state are generic and also capable ofsupporting batch applications, although we leave a directcomparison of the two approaches to future work.

Recent work does combine interactive applicationswith renewables. For example, Krioukov et. al. designa power-proportional cluster targeting interactive appli-cations that responds to power variations by simply de-activating servers and degrading request latency [17].In prior work we propose a blinking abstraction forrenewable-powered clusters, which we have applied tothe distributed caches [26, 28] and distributed file sys-tems [15] commonly used in data centers. However,blinking to support intermittent renewable energy re-quires significant application modifications, while Yankdoes not. iSwitch is perhaps the most closely-relatedwork to Yank [18]. iSwitch assumes a similar design

for integrating renewables into data centers, includingsome servers powered off renewables (specifically windpower) and some powered off the grid. However, iSwitchis more policy-oriented, tracking variations in renew-able power to guide live VM migration between the twoserver pools. In contrast, Yank introduces a new mecha-nism, which iSwitch could use instead of live migration.

7 ConclusionYank introduces the abstraction of a transient server,which may terminate anytime after an advance warning.In this paper, we apply the abstraction to green data cen-ters, where UPSs provides an advance warning, due topower shortfalls from renewables, move transient serverstate to stable servers. Yank fills the void between Re-mus, which requires no advance warning, and live VMmigration, which requires a lengthy advance warning, tocheaply and efficiently support transient servers at largescale. In particular, our results show that a single backupserver is capable of maintaining memory snapshots forup to 15 transient VMs with little performance degrada-tion, which dramatically decreases the cost of providinghigh availability relative to existing solutions.Acknowledgements. We would like to thank our shep-herd, Ratul Mahajan, and the anonymous reviewersfor their valuable comments, as well as Tim Woodand Navin Sharma for their feedback on early ver-sions of this work. We also thank the Remus team,especially Shriram Rajagopalan, for assistance duringearly stages of this project. This research was sup-ported by NSF grants CNS-0855128, CNS-0916972,OCI-1032765, CNS-1117221 and a gift from AT&T.

References[1] S. Akoush, R. Sohan, A. Rice, A.W. Moore, and A. Hop-

per. Predicting the Performance of Virtual Machine Mi-gration. In MASCOTS, August 2010.

[2] B. Aksanli, J. Venkatesh, L. Zhang, and T. Rosing. Uti-lizing Green Energy Prediction to Schedule Mixed Batchand Service Jobs in Data Centers. In HotPower, October2011.

[3] Apple. Apple and the Environment.http://www.apple.com/environment/renewable-energy/,September 2012.

[4] S. Barker, A. Mishra, D. Irwin, E. Cecchet, P. Shenoy,and J. Albrecht. Smart*: An Open Data Set and Tools forEnabling Research in Sustainable Homes. In SustKDD,August 2012.

[5] L. Barroso and U. Holzle. The Datacenter as a Com-puter: An Introduction to the Design of Warehouse-ScaleMachines. Morgan and Claypool Publishers, 2009.

[6] C. Clark, K. Fraser, S. Hand, J. G Hansen, E. Jul,C. Limpach, I. Pratt, and A. Warfield. Live Migrationof Virtual Machines. In NSDI, May 2005.

[7] B. Cully, G. Lefebvre, D. Meyer, M. Feeley, N. Hutchin-son, and A. Warfield. Remus: High Availability via Asyn-chronous Virtual Machine Replication. In NSDI, April2008.

[8] P. Denning. The Working Set Model for Program Behav-ior. CACM, 26(1), January 1983.

[9] DRBD. DRBD: Software Development for High Avail-ability Clusters. http://www.drbd.org/, September 2012.

[10] I. Goiri, W. Katsak, K. Le, T. Nguyen, and R. Bianchini.Parasol and GreenSwitch: Managing Datacenters Pow-ered by Renewable Energy. In ASPLOS, March 2013.

[11] I. Goiri, K. Le, M. Haque, R. Beauchea, T. Nguyen,J. Guitart, J. Torres, and R. Bianchini. GreenSlot:Scheduling Energy Consumption in Green Datacenters.In SC, April 2011.

[12] I. Goiri, K. Le, T. Nguyen, J. Guitart, J. Torres, andR. Bianchini. GreenHadoop: Leveraging Green Energyin Data-Processing Frameworks. In EuroSys, April 2012.

[13] S. Govindan, A. Sivasubramaniam, and B. Urgaonkar.Benefits and Limitations of Tapping into Stored Energyfor Datacenters. In ISCA, June 2011.

[14] S. Govindan, D. Wang, L. Chen, A. Sivasubramaniam,and B. Urgaonkar. Towards Realizing a Low Cost andHighly Available Datacenter Power Infrastructure. InHotPower, October 2011.

[15] D. Irwin, N. Sharma, and P. Shenoy. Towards ContinuousPolicy-driven Demand Response in Data Centers. Com-puter Communications Review, 41(4), October 2011.

[16] Jonathan Koomey. Growth in Data Center Electricity Use2005 to 2010. In Analytics Press, Oakland, California,August 2011.

[17] A. Krioukov, S. Alspaugh, P. Mohan, S. Dawson-Haggerty, D. E. Culler, and R. H. Katz. Design and Evalu-ation of an Energy Agile Computing Cluster. In TechnicalReport UCB/EECS-2012-13, EECS Department, Univer-sity of California, Berkeley, January 2012.

[18] C. Li, A. Qouneh, and T. Li. iSwitch: Coordinating andOptimizing Renewable Energy Powered Server Clusters.In ISCA, June 2012.

[19] H. Liu, C. Xu, H. Jin, J. Gong, and X. Liao. Performanceand Energy Modeling for Live Migration of Virtual Ma-chines. In HPDC, June 2011.

[20] Microsoft. Becoming Carbon Nneutral: How Microsoft isStriving to Become Leaner, Greener, and More Account-able. Microsoft, June 2012.

[21] R. Miller. Facebook Installs Solar Panels at New DataCenter. In Data Center Knowledge, April 16 2011.

[22] U. F. Minhas, S. Rajagopalan, B. Cully, A. Aboulnaga,K. Salem, and A. Warfield. RemusDB: Transparent HighAvailability for Database Systems. In VLDB, August2011.

[23] E. B. Nightingale, K. Veeraraghavan, P. M. Chen, andJ. Flinn. Rethink the Sync. In OSDI, November 2006.

[24] S. Pelley, D. Meisner, P. Zandevakili, T. Wenisch, andJ. Underwood. Power Routing: Dynamic Power Provi-sioning in the Data Center. In ASPLOS, March 2010.

[25] M. Rosenblum and J. Ousterhout. The Design and Imple-mentation of a Log-Structured File System. TOCS, 10(1),February 1992.

[26] N. Sharma, S. Barker, D. Irwin, and P. Shenoy. Blink:Managing Server Clusters on Intermittent Power. In AS-PLOS, March 2011.

[27] N. Sharma, J. Gummeson, D. Irwin, and P. Shenoy.Cloudy Computing: Leveraging Weather Forecasts in En-ergy Harvesting Sensor Systems. In SECON, June 2010.

[28] N. Sharma, D. Krishnappa, D. Irwin, M. Zink, andP. Shenoy. GreenCache: Augmenting Off-the-Grid Cellu-lar Towers with Multimedia Caches. In MMSys, February2013.

[29] N. Sharma, P. Sharma, D. Irwin, and P. Shenoy. Predict-ing Solar Generation from Weather Forecasts Using Ma-chine Learning. In SmartGridComm, October 2011.

[30] N. Tolia, Z. Wang, M. Marwah, C. Bash, P. Ranganathan,and X. Zhu. Delivering Energy Proportionality with NonEnergy-Proportional Systems – Optimizing the Ensem-ble. In HotPower, December 2008.

[31] Y.H. Wan. Long-term Wind Power Variability. Technicalreport, National Renewable Energy Laboratory, January2012.

[32] D. Wang, C. Ren, A. Sivasubramaniam, B. Urgaonkar,and H. Fathy. Energy Storage in Datacenters: What,Where, and How Much? In SIGMETRICS, June 2012.

[33] D. Williams, H. Jamjoom, Y. Liu, and H. Weatherspoon.Overdriver: Handling Memory Overload in an Oversub-scribed Cloud. In VEE, 2011.

[34] T. Wood, A. Lagar-Cavilla, K. K. Ramakrishnan,P. Shenoy, and J. Van der Merwe. PipeCloud: UsingCausality to Overcome Speed-of-Light Delays in Cloud-Based Disaster Recovery. In SOCC, October 2011.

[35] T. Wood, K. Ramakrishnan, P. Shenoy, and J. Van derMerwe. CloudNet: Dynamic Pooling of Cloud Resourcesby Live WAN Migration of Virtual Machines. In VEE,March 2011.