DieCast: Testing Distributed Systems with an Accurate Scale Model

Diwaker Gupta, Kashi V. Vishwanath, and Amin VahdatUniversity of California, San Diego

{dgupta,kvishwanath,vahdat}@cs.ucsd.edu

AbstractLarge-scale network services can consist of tens of thou-sands of machines running thousands of unique soft-ware configurations spread across hundreds of physicalnetworks. Testing such services for complex perfor-mance problems and configuration errors remains a dif-ficult problem. Existing testing techniques, such as sim-ulation or running smaller instances of a service, havelimitations in predicting overall service behavior.

Although technically and economically infeasible atthis time, testing should ideally be performed at the samescale and with the same configuration as the deployedservice. We present DieCast, an approach to scaling net-work services in which we multiplex all of the nodes ina given service configuration as virtual machines (VM)spread across a much smaller number of physical ma-chines in a test harness. CPU, network, and disk are thenaccurately scaled to provide the illusion that each VMmatches a machine from the original service in termsof both available computing resources and communi-cation behavior to remote service nodes. We presentthe architecture and evaluation of a system to supportsuch experimentation and discuss its limitations. Weshow that for a variety of services—including a commer-cial, high-performance, cluster-based file system—andresource utilization levels, DieCast matches the behav-ior of the original service while using a fraction of thephysical resources.

1 IntroductionToday, more and more services are being delivered bycomplex systems consisting of large ensembles of ma-chines spread across multiple physical networks and ge-ographic regions. Economies of scale, incremental scal-ability, and good fault isolation properties have madeclusters the preferred architecture for building planetary-scale services. A single logical request may touch dozensof machines on multiple networks, all providing in-stances of services transparently replicated across mul-tiple machines. Services consisting of tens of thousandsof machines are commonplace [11].

Economic considerations have pushed serviceproviders to a regime where individual service machines

must be made from commodity components—saving anextra $500 per node in a 100,000-node service is critical.Similarly, nodes run commodity operating systems, withonly moderate levels of reliability, and custom-writtenapplications that are often rushed to production becauseof the pressures of “Internet Time.” In this environment,failure is common [24] and it becomes the responsibilityof higher-level software architectures, usually employingcustom monitoring infrastructures and significant serviceand data replication, to mask individual, correlated, andcascading failures from end clients.

One of the primary challenges facing designers ofmodern network services is testing their dynamicallyevolving system architecture. In addition to the sheerscale of the target systems, challenges include: heteroge-neous hardware and software, dynamically changing re-quest patterns, complex component interactions, failureconditions that only manifest under high load [21], theeffects of correlated failures [20], and bottlenecks aris-ing from complex network topologies. Before upgrad-ing any aspect of a networked service—the load balanc-ing/replication scheme, individual software components,the network topology—architects would ideally create anexact copy of the system, modify the single component tobe upgraded, and then subject the entire system to bothhistorical and worst-case workloads. Such testing mustinclude subjecting the system to a variety of controlledfailure and attack scenarios since problems with a par-ticular upgrade will often only be revealed under certainspecific conditions.

Creating an exact copy of a modern networked servicefor testing is often technically challenging and econom-ically infeasible. The architecture of many large-scalenetworked services can be characterized as “controlledchaos,” where it is often impossible to know exactly whatthe hardware, software, and network topology of the sys-tem looks like at any given time. Even when the pre-cise hardware, software and network configuration of thesystem is known, the resources to replicate the produc-tion environment might simply be unavailable, particu-larly for large services. And yet, reliable, low overhead,and economically feasible testing of network services re-mains critical to delivering robust higher-level services.

The goal of this work is to develop a testing method-

1

ology and architecture that can accurately predict the be-havior of modern network services while employing anorder of magnitude less hardware resources. For ex-ample, consider a service consisting of 10,000 hetero-geneous machines, 100 switches, and hundreds of indi-vidual software configurations. We aim to configure asmaller number of machines (e.g., 100-1000 dependingon service characteristics) to emulate the original config-uration as closely as possible and to subject the test in-frastructure to the same workload and failure conditionsas the original service. The performance and failure re-sponse of the test system should closely approximate thereal behavior of the target system. Of course, these goalsare infeasible without giving something up: if it werepossible to capture the complex behavior and overall per-formance of a 10,000 node system on 1,000 nodes, thenthe original system should likely run on 1,000 nodes.

A key insight behind our work is that we can tradetime for system capacity while accurately scaling indi-vidual system components to match the behavior of thetarget infrastructure. We employtime dilation to accu-rately scale the capacity of individual systems by a con-figurable factor [19]. Time dilation fully encapsulatesoperating systems and applications such that the rate atwhich time passes can be modified by a constant factor.A time dilation factor (TDF) of 10 means that for everysecond of real time, all software in a dilated frame be-lieves that time has advanced by only 100 ms. If we wishto subject a target system to a one-hour workload whenscaling the system by a factor of 10, the test would take10 hours of real time. For many testing environments,this is an appropriate tradeoff. Since the passage of timeis slowed down while the rate ofexternal events(suchas network I/O) remains unchanged, the system appearsto have substantially higher processing power and fasternetwork and disk.

In this paper, we present DieCast, a complete envi-ronment for building accurate models of network ser-vices (Section 2). Critically, we run the actual oper-ating systems and application software of some targetenvironment on a fraction of the hardware in that envi-ronment. This work makes the following contributions.First, we extend our original implementation of time di-lation [19] to support fully virtualized as well as paravir-tualized hosts. To support complete system evaluations,our second contribution shows how to extend dilation todisk and CPU (Section 3). In particular, we integratea full disk simulator into the virtual machine monitor(VMM) to consider a range of possible disk architec-tures. Finally, we conduct a detailed system evaluation,quantifying DieCast’s accuracy for a range of services,including a commercial storage system (Sections 4 and5). The goals of this work are ambitious and while wecannot claim to have addressed all of the myriad chal-

lenges associated with testing large-scale network ser-vices (Section 6), we believe that DieCast shows signifi-cant promise as a testing vehicle

2 System ArchitectureWe begin by providing an overview of our approach toscaling a system down to a target test harness. We thendiscuss the individual components of our architecture.

2.1 OverviewFigure 1 gives an overview of our approach. On the left(Figure 1(a)) is an abstract depiction of a network ser-vice. A load balancing switch sits in front of the serviceand redirects requests among a set of front-end HTTPservers. These requests may in turn travel to a middletier of application servers, who may query a storage tierconsisting of databases or network attached storage.

Figure 1(b) shows how a target service can be scaledwith DieCast. We encapsulate all nodes from the origi-nal service in virtual machines and multiplex several ofthese VMs onto physical machines in the test harness.Critically, we employ time dilation in the VMM run-ning on each physical machine to provide the illusionthat each virtual machine has, for example, as much pro-cessing power, disk I/O, and network bandwidth as thecorresponding host in the original configuration despitethe fact that it is sharing underlying resources with otherVMs. DieCast configures VMs to communicate througha network emulator to reproduce the characteristics ofthe original system topology. We then initialize the testsystem using the setup routines of the original systemand subject it to appropriate workloads and fault-loads toevaluate system behavior.

The overall goal is to improve predictive power. Thatis, runs with DieCast on smaller machine configurationsshould accurately predict the performance and fault tol-erance characteristics of some larger production system.In this manner, system developers may experiment withchanges to system architecture, network topology, soft-ware upgrades, and new functionality before deployingthem in production. Successful runs with DieCast shouldimprove confidence that any changes to the target ser-vice will be successfully deployed. Below, we discussthe steps in applying our general approach to applyingDieCast scaling to target systems.

2.2 Choosing the Scaling FactorThe first question to address is the desired scaling fac-tor. One use of DieCast is to reproduce the scale of anoriginal service in a test cluster. Another application isto scale existing test harnesses to achieve more realismthan possible from the raw hardware. For instance, if100 nodes are already available for testing, then DieCastmight be employed to scale to a thousand-node system

(a) Original System (b) Test System

Figure 1: Scaling a network service to the DieCast infrastructure.

with a more complex communication topology. Whilethe DieCast system may still fall short of the scale ofthe original service, it can provide more meaningful ap-proximations under more intense workloads and failureconditions than might have otherwise been possible.

Overall, the goal is to pick the largest scaling factorpossible while still obtaining accurate predictions fromDieCast, since the prediction accuracy will naturally de-grade with increasing scaling factors. This maximumscaling factor depends on the the characteristics of thetarget system. Section 6 highlights the potential limita-tions of DieCast scaling. In general, scaling accuracywill degrade with: i) application sensitivity to the fine-grained timing behavior of external hardware devices;ii) capacity-constrained physical resources; and iii) sys-tem devices not amenable to virtualization. In the firstcategory, application interaction with I/O devices maydepend on the exact timing of requests and responses.Consider for instance a fine-grained parallel applicationthat assumes all remote instances are co-scheduled. ADieCast run may mispredict performance if target nodesare not scheduled at the time of a message transmissionto respond to a blocking read operation. If we could in-terleave at the granularity of individual instructions, thenthis would not be an issue. However, context switchingamong virtual machines means that we must pick timeslices on the order of milliseconds. Second, DieCast can-not scale the capacity of hardware components such asmain memory, processor caches, and disk. Finally, theoriginal service may contain devices such as load bal-ancing switches that are not amenable to virtualization ordilation. Even with these caveats, we have successfullyapplied scaling factors of 10 to a variety of services withnear-perfect accuracy as discussed in Sections 4 and 5.

Of the above limitations to scaling, we consider capac-ity limits for main memory and disk to be most signifi-cant. However, we do not believe this to be a fundamentallimitation. For example, one partial solution is to config-ure the test system with more memory and storage thanthe original system. While this will reduce some of theeconomic benefits of our approach, it will not erase them.For instance, doubling a machine’s memory will not typ-ically double its hardware cost. More importantly, it will

not substantially increase the typically dominant humancost of administering a given test infrastructure becausethe number of required administrators for a given testharness usually grows with the number of machines inthe system rather than with the total memory of the sys-tem.

Looking forward, ongoing research in VMM architec-tures have the potential to reclaim some of the mem-ory [32] and storage overhead [33] associated with multi-plexing VMs on a single physical machine. For instance,four nearly identically configured Linux machines run-ning the same web server will overlap significantly interms of their memory and storage footprints. Similarly,consider an Internet service that replicates content for im-proved capacity and availability. When scaling the ser-vice down, multiple machines from the original configu-ration may be assigned to a single physical machine. AVMM capable of detecting and exploiting available re-dundancy could significantly reduce the incremental stor-age overhead of multiplexing multiple VMs.

2.3 Cataloging the Original System

The next task is to configure the appropriate virtual ma-chine images onto our test infrastructure. Maintaining acatalog of the hardware and software configuration thatcomprises an Internet service is challenging in its ownright. However, for the purposes of this work, we as-sume that such a catalog is available. This catalog wouldconsist of all of the hardware making up the service, thenetwork topology, and the software configuration of eachnode. The software configuration includes the operatingsystem, installed packages and applications, and the ini-tialization sequence run on each node after booting.

The original service software may or may not run ontop of virtual machines. However, given the increasingbenefits of employing virtual machines in data centers forservice configuration and management and the popular-ity of VM-based appliances that are pre-configured to runparticular services [7], we assume that the original ser-vice is in fact VM-based. This assumption is not criticalto our approach but it also partially addresses any base-line performance differential between a node running on

bare hardware in the original service and the same noderunning on a virtual machine in the test system.

2.4 Configuring the Virtual Machines

With an understanding of appropriate scaling factors anda catalog of the original service configuration, DieCastthen configures individual physical machines in the testsystem with multiple VM images reflecting, ideally, aone-to-one map between physical machines in the origi-nal system and virtual machines in the test system. Witha scaling factor of 10, each physical node in the targetsystem would host 10 virtual machines. The mappingfrom physical machines to virtual machines should ac-count for: similarity in software configurations, per-VMmemory and disk requirements and the capacity of thehardware in the original and test system. In general,a solver may be employed to determine a near-optimalmatching [26]. However, given the VM migration capa-bilities of modern VMMs and DieCast’s controlled net-work emulation environment, the actual location of a VMis not as significant as in the original system.

DieCast then configures the VMs such that each VMappears to have resources identical to a physical machinein the original system. Consider a physical machine host-ing 10 VMs. DieCast would run each VM with a scalingfactor of 10, but allocate each VM only 10% of the actualphysical resource. DieCast employs a non-work conserv-ing scheduler to ensure that each virtual machine receivesno more than its allotted share of resources even whenspare capacity is available. Suppose a CPU intensive tasktakes 100 seconds to finish on the original machine. Thesame task would now take 1000 seconds (of real time) ona dilated VM, since it can only use a tenth of the CPU.However, since the VM is running under time dilation,it only perceives that 100 seconds have passed. Thus inthe VMs time frame, resources appear equivalent to theoriginal machine. We only explicitly scale CPU and diskI/O latency on the host; scaling of network I/O happensvia network emulation as described next.

2.5 Network Emulation

The final step in the configuration process is to match thenetwork configuration of the original service using net-work emulation. We configure all VMs in the test sys-tem to route all their communication through our emu-lation environment. Note that DieCast is not tied to anyparticular emulation technology: we have successfullyused DieCast with Dummynet [27], Modelnet [31] andNetem [3] where appropriate.

It is likely that the bisection bandwidth of the origi-nal service topology will be larger than that available inthe test system. Fortunately, time dilation is of signif-icant value here. Convincing a virtual machine scaledby a factor of 10 that it is receiving data at 1 Gbps only

requires forwarding data to it at 100 Mbps. Similarly,it may appear that latencies in an original cluster-basedservice may be low enough that the additional softwareforwarding overhead associated with the emulation en-vironment could make it difficult to match the latenciesin the original network. To our advantage, maintainingaccurate latency with time dilation actually requiresin-creasingthe real time delay of a given packet; e.g., a 100µs delay network link in the original network should bedelayed by 1 ms when dilating by a factor of 10.

Note that the scaling factor need not match the TDF.For example, if the original network topology is solarge/fast that even with a TDF of 10 the network emu-lator is unable to keep up, it is possible to employ a timedilation factor of 20 while maintaining a scaling factor of10. In such a scenario, there would still on average be10 virtual machines multiplexed onto each physical ma-chine, however the VMM scheduler would allocate only5% of the physical machine’s resources to individual ma-chines (meaning that 50% of CPU resources will go idle).The TDF of 20, however, would deliver additional capac-ity to the network emulation infrastructure to match thecharacteristics of the original system.

2.6 Workload GenerationOnce DieCast has prepared the test system to beresourceequivalentto the original system, we can subject it toan appropriate workload. These workloads will in gen-eral be application-specific. For instance, Monkey [15]shows how to replay a measured TCP request stream sentto a large-scale network service. For this work, we useapplication-specific workload generators where availableand in other cases write our own workload generators thatboth capture normal behavior as well as stress the serviceunder extreme conditions.

To maintain a target scaling factor, clients should alsoideally run in DieCast-scaled virtual machines. This ap-proach has the added benefit of allowing us to subject atest service to a high level of perceived-load using rela-tively few resources. Thus, DieCast scales not only thecapacity of the test harness but also the workload gener-ation infrastructure.

3 ImplementationWe have implemented DieCast support on several ver-sions of Xen [10]: v2.0.7, v3.0.4, and v3.1 (both par-avirtualized and fully virtualized VMs). Here we focuson the Xen 3.1 implementation. We begin with a briefoverview of time dilation [19] and then describe the newfeatures required to support DieCast.

3.1 Time DilationCritical to time dilation is a VMM’s ability to modify theperception of time within a guest OS. Fortunately, most

VMMs already have this functionality, for example, be-cause a guest OS may develop a backlog of “lost ticks”if it is not scheduled on the physical processor when itis due to receive a timer interrupt. Since the guest OSrunning in a VM does not run continuously, VMMs peri-odically synchronize the guest OS time with the physicalmachine’s clock. The only requirement for a VMM tosupport time dilation is this ability to modify the VM’sperception of time. In fact, as we demonstrate in Sec-tion 5, the concept of time dilation can be ported to other(non-virtualized) environments.

Operating systems employ a variety of time sourcesto keep track of time, including timer interrupts (e.g., theProgrammable Interrupt Timer or PIT), specialized coun-ters (e.g., the TSC on Intel platforms) and external timesources such as NTP. Time dilation works by interceptingthe various time sources and scaling them appropriatelyto fully encapsulate the OS in its own time frame.

Our original modifications to Xen for paravirtualizedhosts [19] therefore appropriately scale time values ex-posed to the VM by the hypervisor. Xen exposes twonotions of time to VMs. Real time is the number ofnanoseconds since boot, and wall clock time is the tradi-tional Unix time since epoch. While Xen allows the guestOS to maintain and update its own notion of time via anexternal time source (such as NTP), the guest OS oftenrelies solely on Xen to maintain accurate time. Real andwall clock time pass between the Xen hypervisor and theguest operating system via a shared data structure. Di-lation uses a per-domain TDF variable to appropriatelyscale real time and wall clock time. It also scales the fre-quency of timer interrupts delivered to a guest OS sincethese timer interrupts often drive the internal time keep-ing of a guest. Given these modifications to Xen, ourearlier work showed that network dilation matches undi-lated baselines for complex per-flow TCP behavior in avariety of scenarios [19].

3.2 Support for OS diversity

Our original time dilation implementation only workedwith paravirtualized machines, with two major draw-backs: it supported only Linux as the guest OS, andthe guest kernel required modifications. Generalizingto other platforms would have required code modifi-cations to the respective OS. To be widely applicable,DieCast must support a variety of operating systems.

To address these limitations, we ported time dilation tosupportfully virtualized(FV) VMs, enabling DieCast tosupport unmodified OS images. Note that FV VMs re-quire platforms with hardware support for virtualization,such as Intel VT or AMD SVM. While Xen support forfully virtualized VMs differs significantly from the par-avirtualized VM support in several key areas such as I/Oemulation, access to hardware registers, and time man-

agement, the general idea behind the implementation re-mains the same: we want to intercept all sources of timeand scale them.

In particular, our implementation scales the PIT, theTSC register (on x86), the RTC (Real Time Clock), theACPI power management timer and the High Perfor-mance Event Timer (HPET). As in the original imple-mentation, we also scale the number of timer interruptsdelivered to a fully virtualized guest. We allow each VMto run with an independent scaling factor. Note, how-ever, that the scaling factor is fixed for the life time of aVM—it can not be changed at run time.

3.3 Scaling Disk I/O and CPU

Time dilation as described in [19] did not scale disk per-formance, making it unsuitable for services that performsignificant disk I/O. Ideally, we would scale individualdisk requests at the disk controller layer. The complexityof modern drive architectures, particularly the fact thatmuch low level functionality is implemented in firmware,makes such implementations challenging. Note that sim-ply delaying requests in the device driver is not sufficient,since disk controllers may re-order and batch requests forefficiency. On the other hand, functionality embedded inhardware or firmware is difficult to instrument and mod-ify. Further complicating matters are the different I/Omodels in Xen: one for paravirtualized (PV) VMs andone for fully virtualized (FV) VMs. DieCast providesmechanisms to scale disk I/O for both models.

For FV VMs, DieCast integrates a highly accurate andefficient disk system simulator — Disksim [17] — whichgives us a good trade-off between realism and accuracy.Figure 2(a) depicts our integration of DiskSim into thefully virtualized I/O model: for each VM, a dedicateduser space process (ioemu) in Domain-0 performs I/Oemulation by exposing a “virtual disk” to the VM (theguest OS is unaware that a real disk is not present). Aspecial file in Domain-0 serves as the backend storagefor the VM’s disk. To allowioemu to interact withDiskSim, we wrote a wrapper around the simulator forinter-process communication.

After servicing each request (but before returning),ioemu forwards the request to Disksim, which then re-turns the time,rt, the request would have taken in itssimulated disk. Since we are effectively layering a soft-ware disk on top ofioemu, each request should ideallytake exactly timert in the VM’s time frame, ortdf ∗ rt

in real time. If delay is the amount by which this re-quest is delayed, the total time spent inioemu becomesdelay + dt + st, wherest is the time taken toactuallyserve the request (Disksim only simulates I/O character-istics, it does not deal with the actual disk content) anddt

is the time taken to invoke Disksim itself. The requireddelay is then(tdf ∗ rt) − dt − st.

(a) I/O Model for FV VMs (b) I/O Model for PV VMs

0 2 4 6 8 10Time Dilation Factor

0

500

1000

1500

2000

Dis

k t

hro

ug

hp

ut

(kB

/s)

CPU and Disk unscaledCPU scaled (old)Disk and CPU scaledDisk and CPU scaled (improved)

(c) DBench throughput under Disksim

Figure 2: Scaling Disk I/O

The architecture of Disksim, however, is not amenableto integration with the PV I/O model (Figure 2(b)). Inthis “split I/O” model, a front-end driver in the VM(blkfront) forwards requests to a back-end driver inDomain-0 (blkback), which are then serviced by thereal disk device driver. Thus PV I/O is largely a kernelactivity, while Disksim runs entirely in user-space. Fur-ther, a separate Disksim process would be required foreach simulated disk, whereas there is a single back-enddriver for all VMs.

For these reasons, for PV VMs, we inject the appropri-ate delays in theblkfront driver. This approach hasthe additional advantage of containing the side effects ofsuch delays to individual VMs —blkback can con-tinue processing other requests as usual. Further, it elimi-nates the need to modify disk-specific drivers in Domain-0. We emphasize that this is functionally equivalent toper-request scaling in Disksim: the key difference is thatscaling in Disksim is much closer to the (simulated) hard-ware. Overall our implementation of disk scaling for PVVM’s is simpler though less accurate and somewhat lessflexible since it requires the disk subsystem in the testinghardware to match the configuration in the target system.

We have validated both our implementations usingseveral micro-benchmarks. For brevity, we only describeone of them here. We run DBench [29] — a popular hard-drive and file-system benchmark — under different dila-tion factors and plot the reported throughput. Figure 2(c)shows the results for the FV I/O model with Disksim in-tegration (results for the PV implementation can be foundin a separate technical report [18]). Ideally, the through-put should remain constant as a function of the dilationfactor. We first run the benchmark without scaling diskI/O or CPU, and we can see that the reported throughputincreases almost linearly, an undesirable behavior. Next,we repeat the experiment and scale the CPU alone (thus,at TDF 10 the VM only receives 10% of the CPU). Whilethe increase is no longer linear, in the absence of diskdilation it is still significantly higher than the expectedvalue. Finally, with disk dilation in place we can see thatthe throughput closely tracks the expected value.

However, as the TDF increases, we start to see somedivergence. After further investigation, we found thatthis deviation results from the way we scaled the CPU.Recall that we scale the CPU by bounding the amountof CPU available to each VM. Initially, we simply usedXen’s Credit scheduler to allocate an appropriate fractionof CPU resources to each VM in non-work conservingmode. However, simply scaling the CPU does not governhow those CPU cycles are distributed across time. Withthe original Credit scheduler, if a VM does not consumeits full timeslice, it can be scheduled again in subsequenttimeslices. For instance, if a VM is set to be dilated bya factor of 10 and if it consumes less than 10% of theCPU in each time slice, then it will run inevery timeslice, since in aggregate it never consumes more than itshard bound of 10% of the CPU. This potential to run con-tinuously distorts the performance of I/O-bound applica-tions under dilation, and in particular they’ll have a dif-ferent timing distribution than they would in the real timeframe. This distortion increases with increasing TDF.Thus, we found that, for some workloads, we may ac-tually wish to enforce that the VM’s CPU consumptionshould be moreuniformlyenforced across time.

We modified the Credit CPU scheduler in Xen to sup-port this mode of operation as follows: if a VM runs forthe entire duration of its time slice, we ensure that it doesnot get scheduled for the next(tdf − 1) time slices. If aVM voluntarily yields the CPU or is pre-empted beforeits time slice expires, itmay be re-scheduled in a sub-sequent time slice. However, as soon as it consumes acumulative total of a time slice’s worth of run time (car-ried over from the previous time it was descheduled), itwill be pre-empted and not allowed to run for another(tdf − 1) time slices. The final line in figure 2(c) showsthe results of the DBench benchmark with using thismodified scheduler. As we can see, the throughput re-mains consistent even at higher TDFs. Note that unlikein this benchmark, DieCast typically runs multiple VMsper machine, in which case this “spreading” of CPU cy-cles occurs naturally as VMs compete for CPU.

4 EvaluationWe seek to answer the following questions with respectto DieCast-scaling: i) Can we configure a smaller num-ber of physical machines to match the CPU capacity,complex network topology, and I/O rates of a larger ser-vice? ii) How well does the performance of a scaled ser-vice running on fewer resources match the performanceof a baseline service running with more resources? weconsider three different systems: i) BitTorrent, a popularpeer-to-peer file sharing program; ii) RUBiS, an auctionservice prototyped after eBay; and iii) Isaac, our config-urable network three-tier service that allows us to gener-ate a range of workload scenarios.

4.1 Methodology

To evaluate DieCast for a given system, we first estab-lish the baseline performance: this involves determiningthe configuration(s) of interest, fixing the workload, andbenchmarking the performance. We then scale the sys-tem down by an order of magnitude and compare theDieCast performance to the baseline. While we have ex-tensively evaluated evaluated DieCast implementationsfor several versions of Xen, we only present the resultsfor the Xen 3.1 implementation here. Detailed evaluationfor Xen 3.0.4 can be found in our technical report [18].

Each physical machine in our testbed is a dual-core2.3GHz Intel Xeon with 4GB RAM. Note that since theDisksim integration only works with fully virtualizedVMs, for a fair evaluation it isrequired that even thebaseline system run on VMs—ideally the baseline wouldbe run on physical machines directly (for the paravirtual-ized setup, we do have evaluation with physical machinesas the baseline. We refer the reader to [18] for details).We configure Disksim to emulate a Seagate ST3217 diskdrive. For the baseline, Disksim runs as usual (no re-quests are scaled) and with DieCast, we scale each re-quest as described in Section 3.3.

We configure each virtual machine with 256MB RAMand run Debian Etch on Linux 2.6.17. Unless otherwisestated, the baseline configuration consists of 40 physicalmachines hosting a single VM each. We then comparethe performance characeteristics to runs with DieCast onfour physical machines hosting 10 VMs each, scaled bya factor of 10. We use Modelnet for the network emu-lation, and appropriately scale the link characteristics forDieCast. For allocating CPU, we use our modified CreditCPU scheduler as described in Section 3.3.

4.2 BitTorrent

We begin by using DieCast to evaluate BitTorrent [1] —a popular P2P application. For our baseline experiments,we run BitTorrent (version 3.4.2) on a total of 40 virtualmachines. We configure the machines to communicate

across a ModelNet-emulated dumbbell topology (Figure3), with varying bandwidth and latency values for the ac-cess link (A) from each client to the dumbbell and thedumbbell link itself (C). We vary the total number ofclients, the file size, the network topology, and the ver-sion of the BitTorrent software. We use the distributionof file download times across all clients as the metric forcomparing performance. The aim here is to observe howclosely DieCast-scaled experiments reproduce behaviorof the baseline case for a variety of scenarios.

The first experiment establishes the baseline where wecompare different configurations of BitTorrent sharing afile across a 10Mbps dumbbell link and constrained ac-cess links of 10Mbps. All links have a one-way latencyof 5ms. We run a total of 40 clients (with half on eachside of the dumbbell). Figure 5 plots the cumulative dis-tribution of transfer times across all clients for differentfile sizes (10MB and 50MB). We show the baseline caseusing solid lines and use dashed lines to represent theDieCast-scaled case. With DieCast scaling, the distribu-tion of download times closely matches the behavior ofthe original system. For instance, well-connected clientson the same side of the dumbbell as the randomly cho-sen seeder finish more quickly than the clients that mustcompete for scarce resources across the dumbbell.

Having established a reasonable baseline, we next con-sider sensitivity to changing system configurations. Wefirst vary the network topology by leaving the dumbbelllink unconstrained (1 Gbps) with results in Figure 5. Thegraph shows the effect of removing the bottleneck on thefinish times compared to the constrained dumbbell-linkcase for the 50-MB file: all clients finish within a smalltime difference of each other as shown by the middle pairof curves.

Next, we consider the effect of varying the total num-ber of clients. Using the topology from the baseline ex-periment we repeat the experiments for 80 and 200 simul-taneous BitTorrent clients. Figure 6 shows the results.The curves for the baseline and DieCast-scaled versionsalmost completely overlap each other for 80 clients (leftpair of curves) and show minor deviation from each otherfor 200 clients (right pair of curves). Note that with 200clients, the bandwidth contention increases to the pointwhere the dumbbell bottleneck becomes less important.

Finally, we consider an experiment that demonstratesthe flexibility of DieCast to reproduce system perfor-mance under a variety of resource configurations start-ing with the same baseline. Figure 7 shows that in addi-tion to matching 1:10 scaling using 4 physical machineshosting 10 VMs each, we can also match an alternateconfiguration of 8 physical machines, hosting five VMseach with a dilation factor of five. This demonstrates thateven if it is necessary to vary the number of physical ma-chines available for testing, it may still be possible to find

Figure 3: Topology for BitTorrent experiments. Figure 4: RUBiS Setup.

0 20 40 60 80 100 120 140 160Time to complete since start of experiment (s)

0.0

0.2

0.4

0.6

0.8

1.0

Cu

mu

lati

ve F

racti

on

10 MB 50 MB

1-Gbps Bottleneck

Baseline Diecast

Figure 5: Performance with varying file sizes.

0 100 200 300 400 500Time to complete since start of experiment (s)

0.0

0.2

0.4

0.6

0.8

1.0

Cu

mu

lati

ve F

racti

on

80 clients

200 clients

Baseline DieCast

Figure 6: Varying #clients.

0 20 40 60 80 100 120 140 160 180Time to complete since start of experiment (s)

0.0

0.2

0.4

0.6

0.8

1.0

Cu

mu

lati

ve F

racti

on

No DieCast

Baseline DieCast TDF 10 DieCast TDF 5 No DieCast

Figure 7: Different configurations.

an appropriate scaling factor to match performance char-acteristics. This graph also has a fourth curve, labeled“No DieCast”, corresponding to running the experimentwith 40 VMs on four physical machines, each with a di-lation factor of 1—disk and network arenot scaled (thusmatch the baseline configuration), and all VMs are allo-cated equal shares of the CPU. This corresponds to theapproach of simply multiplexing a number of virtual ma-chines on physical machines without using DieCast. Thegraph shows that the behavior of the system under such anave approach varies widely from actual behavior.

4.3 RUBiSNext, we investigate DieCast’s ability to scale a fullyfunctional Internet service. We use RUBiS [6]—an auc-tion site prototype designed to evaluate scalability andapplication server performance. RUBiS has been usedby other researchers to approximate realistic Internet Ser-vices [12–14].

We use the PHP implementation of RUBiS runningApache as the web server and MySQL as the database.For consistent results, we re-create the database and pre-populate it with 100,000 users and items before each ex-periment. We use the default read-write transaction ta-ble for the workload that exercises all aspects of the sys-tem such as adding new items, placing bids, adding com-ments, viewing and browsing the database. The RUBiSworkload generators warm up for 60 seconds, followedby a session run time of 600 seconds and ramp down for60 seconds.

We emulate a topology of 40 nodes consisting of 8database servers, 16 web servers and 16 workload gen-erators as shown in Figure 4. A 100 Mbps networklink connects two replicas of the service spread acrossthe wide-area at two sites. Within a site, 1 Gbps linksconnect all components. For reliability, half of the webservers at each site use the database servers in the othersite. There is one load generator per web server and allload generators share a 100 Mbps access link. Each sys-tem component (servers, workload generators) runs in itsown Xen VM.

We now evaluate DieCast’s ability to predict the be-havior of this RUBiS configuration using fewer re-sources. Figures 8(a) and 8(b) compare the baselineperformance with the scaled system for overall systemthroughput and average response time (across all client-webserver combinations) on the y-axis as a function ofnumber of simultaneous clients (offered load) on the x-axis. In both cases, the performance of the scaled ser-vice closely tracks that of the baseline. We also show theperformance for the “No DieCast” configuration: reg-ular VM multiplexing with no DieCast-scaling. With-out DieCast to offset the resource contention, the aggre-gate throughput drops with a substantial increase in re-sponse times. Interestingly, for one of our initial tests, weran with an unintended mis-configuration of the RUBiSdatabase: the workload had commenting-related opera-tions enabled, but the relevant tables were missing fromthe database. This led to an approximately 25% error rate

0 1000 2000 3000 4000 5000 6000 7000

Total system load (User sessions)

0

10000

20000

30000

40000

50000

60000

Ag

gre

gate

Th

rou

gh

pu

t (r

eq

uests

/min

)

BaselineDieCastNo DieCast

(a) Throughput

0 1000 2000 3000 4000 5000 6000 7000

Total system load (User sessions)

0

1000

2000

3000

4000

5000

6000

7000

8000

Resp

on

se t

ime (

ms)

BaselineDieCastNo DieCast

(b) Response Time

Figure 8: Comparing RUBiS application performance: Baselinevs. DieCast.Figure 10:Architecture of Isaac.

0 200 400 600 800 1000

Time since start of experiment (s)

0

20

40

60

80

100

CP

U u

sed

(%

)

DB server

0

20

40

60

80

100

CP

U u

sed

(%

)

Web server

0

20

40

60

80

100

CP

U u

sed

(%

)

Client

BaselineDieCast

(a) CPU profile

0 200 400 600 800 1000 1200Time since start of experiment (s)

0

20

40

60

80

100

Mem

ory

uti

lizati

on

(%

)

DB server

Web server

Client

BaselineDieCast

(b) Memory profile

0 10 20 30 40 50 60 70 80 90Hop ID

10-1

100

101

102

103

104

Data

tra

nsfe

rred

(M

B)

BaselineDieCast

(c) Network profile

Figure 9: Comparing resource utilization for RUBiS: DieCastcan accurately emulate the baseline system behavior.

with similar timings in the responses to clients in both thebaseline and DieCast configurations. These types of con-figuration errors are one example of the types of testingthat we wish to enable with DieCast.

Next, Figures 9(a) and 9(b) compare CPU and mem-ory utilizations for both the scaled and unscaled experi-ments as a function of time for the case of 4800 simul-taneous user sessions: we pick one node of each type(DB server, Web server, load generator) at random fromthe baseline, and use the same three nodes for compari-son with DieCast. One important question is whether theaverage performance results in earlier figures hide signif-icant incongruities in per-request performance. Here, wesee that resource utilization in the DieCast-scaled exper-iments closely tracks the utilization in the baseline on aper-node and per-tier (client, web server, database) ba-sis. Similarly, Figure 9(c) compares the network utiliza-tion of individual links in the topology for the baselineand DieCast-scaled experiment. We sort the links by the

amount of data transferred per link in the baseline case.This graph demonstrates that DieCast closely tracks andreproduces variability in network utilization for varioushops in the topology. For instance, hops 86 and 87 in thefigure correspond to access links of clients and show themaximum utilization, whereas individual access links ofWebservers are moderately loaded.

4.4 Exploring DieCast Accuracy

While we were encouraged by DieCast’s ability to scaleRUBiS and BitTorrent, they represent only a few pointsin the large space of possible network service configura-tions, for instance, in terms of the ratios of computationto network communication to disk I/O. Hence, we builtIsaac, a configurable multi-tier network service to stressthe DieCast methodology on a range of possible config-urations. Figure 10 shows Isaac’s architecture. Requestsoriginating from a client (C) travel to a unique front endserver (FS) via a load balancer (LB). The FS makes

0 100 200 300 400 500 600 700Time since start of experiment (s)

0.0

0.2

0.4

0.6

0.8

1.0F

racti

on

of

req

uests

com

ple

ted

BaselineDieCast

Figure 11: Request completion time.

DB AS FSDifferent Tiers

0

20

40

60

80

100

Tim

e S

pen

t in

Each

Tie

r (%

)

Figure 12: Tier-breakdown.

0 200 400 600 800 1000 1200 1400 1600Time since start of experiment (s)

0.0

0.2

0.4

0.6

0.8

1.0

Fra

cti

on

of

req

uests

com

ple

ted

CPU stress

100-KB writes

No DieCast stress-CPU

No DieCast stress-DB

BaselineDieCast

Figure 13: Stressing DB/CPU.

a number of calls to other services through applicationservers (AS). These application servers in turn may is-sue read and write calls to a database back end (DB)before building a response and transmitting it back to thefront end server, which finally responds to the client.

Isaac is written in Python and allows configuring theservice to a given interconnect topology, computation,communication, and I/O pattern. A configuration de-scribes, on a per request class basis, the computation,communication, and I/O characteristics across multipleservice tiers. In this manner, we can configure experi-ments to stress different aspects of a service and to in-dependently push the system to capacity along multipledimensions. We use MySQL for the database tier to re-flect a realistic transactional storage tier.

For our first experiment, we configure Isaac with fourDBs, four ASs, four FSs and28 clients. The clients gen-erate requests, wait for responses, and sleep for sometime before generating new requests. Each client gener-ates20 requests and each such request touches five ASs(randomly selected at run time) after going through theFS. Each request from the AS involves10 reads from and2 writes to a database each of size1KB. The databaseserver is also chosen randomly at runtime. Upon com-pleting its database queries, each AS computes500 SHA-1 hashes of the response before sending it back to the FS.Each FS then collects responses from all five AS’s and fi-nally computes 5,000 SHA-1 hashes on the concatenatedresults before replying to the client. In later experiments,we vary both the amount of computation and I/O to quan-tify sensitivity to varying resource bottlenecks

We perform this40-node experiment both with andwithout DieCast. For brevity, we do not show the re-sults of initial tests validating DieCast accuracy (in allcases, performance matched closely in both the dilatedand baseline case). Rather, we run a more complex ex-periment where a subset of the machines fail and thenrecover. Our goal is to show that DieCast can accuratelymatch application performance before the failure occurs,during the failure scenario, and the application’s recoverybehavior. After 200 seconds, we fail half of the databaseservers (chosen at random) by stopping MySQL servers

on the corresponding nodes. As a result, client requestsaccessing failed databases will not complete, slowing therate of completed requests. After one minute of down-time, we restart the MySQL server and soon after weexpect to see the request completion rate to regain itsoriginal value. Figure 11 shows fraction of requests com-pleted on the Y-axis as a function of time since the start ofthe experiment on the X-axis. DieCast closely matchesthe baseline application behavior with a dilation factorof 10. We also compare the percentage of time spentin each of the three tiers of Isaac averaged across all re-quests. Figure 12 shows that in addition to the end-to-endresponse time, DieCast closely tracks the system behav-ior on a per-tier basis.

Encouraged by the results of the previous experi-ment, we next attempt to saturate individual compo-nents of Isaac to explore the limits of DieCast’s accuracy.First, we evaluate DieCast’s ability to scale network ser-vices when database access dominates per-request ser-vice time. Figure 13 shows the completion time for re-quests, where each service issues a 100-KB (rather than1-KB) write to the database with all other parameters re-maining the same. This amounts to a total of 1 MB ofdatabase writes for every request from a client. Even forthese larger data volumes, DieCast faithfully reproducessystem performance. While for this workload, we areable to maintain good accuracy, the evaluation of disk di-lation summarized in Figure 2(c) suggests that there willcertainly be points where disk dilation inaccuracy willaffect overall DieCast accuracy.

Next, we evaluate DieCast accuracy when one ofthe components in our architecture saturates the CPU.Specifically, we configure our front-end servers such thatprior to sending each response to the client, they computeSHA-1 hashes of the response 500,000 times to artifi-cially saturate the CPU of this tier. The results of this ex-periment too are shown in Figure 13. We are encouragedoverall as the system does not significantly diverge evento the point of CPU saturation. For instance, the CPUutilization for nodes hosting the FS in this experimentvaried from50− 80% for the duration of the experimentand even under such conditions DieCast closely matched

the baseline system performance. The “No DieCast”lines plot the performance of the stress-DB and stress-CPU configurations with regular VM multiplexing with-out DieCast-scaling. As with BitTorrent and RUBiS, wesee that without DieCast, the test infrastructure fails topredict the performance of the baseline system.

5 Commercial System EvaluationWhile we were encouraged by DieCast’s accuracy for theapplications we considered in Section 4, all of the ex-periments were designed by DieCast authors and werelargely academic in nature. To understand the generalityof our system, we consider its applicability to a large-scale commercial system.

Panasas [4] builds scalable storage systems target-ing Linux cluster computing environments. It has sup-plied solutions to several government agencies, oil andgas companies, media companies and several commer-cial HPC enterprises. A core component of Panasas’sproducts is the PanFS parallel filesystem (henceforth re-ferred to as PanFS): an object-based cluster filesystemthat presents a single, cache coherent unified namespaceto clients.

To meet customer requirements, Panasas must ensureits systems can deliver appropriate performance under arange of client access patterns. Unfortunately, it is of-ten impossible to create a test environment that reflectsthe setup at a customer site. Since Panasas has severalcustomers with very large super-computing clusters andlimited test infrastructure at its disposal, its ability toper-form testing at scale is severely restricted by hardwareavailability; exactly the type of situation DieCast tar-gets. For example, the Los Alamos National Lab has de-ployed PanFS with its Roadrunner peta-scale super com-puter [5]. The Roadrunner system is designed to delivera sustained performance level of one petaflop at an esti-mated cost of $90 million. Because of the tremendousscale and cost, Panasas cannot replicate this computingenvironment for testing purposes.

Porting Time Dilation. In evaluating our ability to ap-ply DieCast to PanFS, we encountered one primary limi-tation. PanFS clients use a Linux kernel module to com-municate with the PanFS server. The client-side coderuns on recent versions of Xen , and hence, DieCast sup-ported them with no modifications. However, the PanFSserver runs in a custom operating system derived from anolder version of FreeBSD that does not support Xen. Thesignificant modifications to the base FreeBSD operatingsystem made it impossible to port PanFS to a more re-cent version of FreeBSD that does support Xen. Ideally,it would be possible to simply encapsulate the PanFSserver in a fully virtualized Xen VM. However, recallthat this requires virtualization support in the processor

64K 1M 2M 4M 16MBlock sizes

50

100

150

200

250

300

350

400

450

Th

rou

gh

pu

t (M

B/s

)

MPI-IO

IOZone

Baseline ReadBaseline WriteDieCast ReadDieCast Write

Figure 14: Validating DieCast on PanFS.

which was unavailable in the hardware Panasas was us-ing. Even if we had the hardware, Xen did not supportFreeBSD on FV VMs until recently due to a well knownbug [2]. Thus, unfortunately we could not easily employthe existing time dilation techniques with PanFS on theserver side. However, since we believe DieCast conceptsare general and not restricted to Xen, we took this oppor-tunity to explore whether we could modify the PanFS OSto support DieCast, without any virtualization support.

To implement time dilation in the PanFS kernel, wescale the various time sources , and consequently, thewall clock. The TDF can be specified at boot time asa kernel parameter. As before, we need to scale downresources available to PanFS such that its perceived ca-pacity matches the baseline.

For scaling the network, we use Dummynet [27],which ships as part of the PanFS OS. However, there wasno mechanism for limiting the CPU available to the OS,or to slow the disk. The PanFS OS does not support nonwork-conserving CPU allocation. Further, simply modi-fying the CPU scheduler for user processes is insufficientbecause it would not throttle the rate of kernel process-ing. For CPU dilation, we had to modify the kernel asfollows. We created a CPU-bound task, (idle), in thekernel and we statically assigned it the highest schedul-ing priority. We scale the CPU by maintaining the re-quired ratio between the run times of theidle task andall remaining tasks. If theidle task consumes suffi-cient CPU, it is removed from the run queue and the reg-ular CPU scheduler kicks in. If not, the scheduler alwayspicks theidle task because of its priority.

For disk dilation, we were faced by the complicationthat multiple hardware and software components interactin PanFS to service clients. For performance, there areseveral parallel data paths and many operations are eitherasynchronous or cached. Accurately implementing diskdilation would require accounting for all of the possiblecode paths as well as modeling the disk drives with highfidelity. In an ideal implementation, if the physical ser-vice time for a disk request iss and the TDF ist, then therequest should be delayed by time(t − 1)s such that thetotal physical service time becomest × s, which under

dilation would be perceived as the desired value ofs.Unfortunately, the Panasas operating system only pro-

vides coarse-grained kernel timers. Consequently, sleepcalls with small durations tend to be inaccurate. Usinga number of micro-benchmarks, we determined that thesmallest sleep interval that could be accurately imple-mented in the PanFS operating system was 1 ms.

This limitation affects the way disk dilation can be im-plemented. For I/O intensive workloads, the rate of diskrequests is high. At the same time, the service time ofeach request is relatively modest. In this case, delayingeach request individually is not an option, since the over-head of invoking sleep dominates the injected delay andgives unexpectedly large slowdowns. Thus, we chose toaggregate delays across some number of requests whoseservice time sums to more than 1 ms and periodically in-ject delays rather than injecting a delay for each request.Another practical limitation is that it is often difficult toaccurately bound the service time of a disk request. Thisis a result of the various I/O paths that exist: requests canbe synchronous or asynchronous, they can be servicedfrom the cache or not and so on.

While we realize that this implementation is imperfect,it works well in practice and can be automatically tunedfor each workload. A perfect implementation would haveto accurately model the low level disk behavior and im-prove the accuracy of the kernel sleep function. Becauseoperating systems and hardware will increasingly sup-port native virtualization, we feel that our simple disk di-lation implementation targeting individual PanFS work-loads is reasonable in practice to validate our approach.

Validation We first wish to establish DieCast accuracyby running experiments on bare hardware and comparingthem against DieCast-scaled virtual machines. We startby setting up a storage system consisting of an PanFSserver with 20 disks of capacity 250GB each (5TB totalstorage). We evaluate two benchmarks from the stan-dard bandwidth test suite used by Panasas. The firstbenchmark involves 10 clients (each on a separate ma-chine) running IOZone [23]. The second benchmark usesthe Message Passing Interface (MPI) across 100 clients(again, on separate machines) [28].

For DieCast scaling, we repeat the experiment with ourmodifications to the PanFS server configured to enforce adilation factor of 10. Thus, we allocate 10% of the CPUto the server and dilate the network using Dummynet to10% of the physical bandwidth and 10 times the latency(to preserve the bandwidth-delay product). On the clientside, we have all clients running in separate virtual ma-chines (10 VMs per physical machine), each receiving10% of the CPU with a dilation factor of 10.

Figure 14 plots the aggregate client throughput forboth experiments on the y-axis as a function of thedata block size on the x-axis. Circles mark the read

AggregateThroughput

Number of clients

10 250 1000

Write 370 MB/s 403 MB/s 398 MB/sRead 402 MB/s 483 MB/s 424 MB/s

Table 1: Aggregate read/write throughputs from the IOZone benchmarkwith block size 16M. PanFS performance scales gracefully with largerclient populations.

throughput while triangles mark write throughput. Weuse solid lines for the baseline and dashed lines for theDieCast-scaled configuration. For both reads and writes,DieCast closely follows baseline performance, never di-verging by more than 5% even for unusually large blocksizes.

Scaling With sufficient faith in the ability of DieCast toreproduce performance for real-world application work-loads we next aim to push the scale of the experimentbeyond what Panasas can easily achieve with their exist-ing infrastructure.

We are interested in the scalability of PanFS as weincrease the number of clients by two orders of magni-tude. To achieve this, we design an experiment similarto the one above, but this time we fix the block size at16MB and vary the number of clients. We use 10 VMseach on 25 physical machines to support 250 clients torun the IOZone benchmark. We further scale the exper-iment by using 10 VMs each on 100 physical machinesto go up to 1000 clients. In each case, all VMs are run-ning at a TDF of 10. The PanFS server also runs at aTDF of 10 and all resources (CPU, network, disk) arescaled appropriately. Table 1 shows that the performanceof PanFS with increasing client population. Interestingly,we find relatively little increase in throughput as we in-crease the client population. Upon investigating further,we found that a single PanFS server configuration is lim-ited to 4 Gb/s (500 MB/s) of aggregate bisection band-width between the servers and clients (including any IPand filesystem overhead). While our network emulationaccurately reflected this bottleneck, we did not catch thebottleneck until we ran our experiments. We leave a per-formance evaluation when removing this bottleneck tofuture work.

We would like to emphasize that prior to our experi-ment, Panasas had been unable to perform experiments atthis scale. This is in part due to the fact that such a largenumber of machines might not be available at any giventime for a single experiment. Further, even if machinesare available, blocking a large number of machines re-sults in significant resource contention because severalother smaller experiments are then blocked on avail-ability of resources. Our experiments demonstrate thatDieCast can leverage existing resources to work around

these types of problems.

6 DieCast Usage ScenariosIn this section, we discuss DieCast’s applicability andlimitations for testing large-scale network services in avariety of environments.

DieCast aims to reproduce the performance of an orig-inal system configuration and is well suited for predict-ing the behavior of the system under a variety of work-loads. Further, because the test system can be subject toa variety of realistic and projected client access patterns,DieCast may be employed to verify that the system canmaintain the terms of Service Level Agreements (SLA).

It runs in a controlled and partially emulated networkenvironment. Thus, it is relatively straightforward to con-sider the effects of revamping a service’s network topol-ogy (e.g., to evaluate whether an upgrade can alleviatea communication bottleneck). DieCast can also system-atically subject the system to failure scenarios. For ex-ample, system architects may develop a suite of fault-loads to determine how well a service maintains responsetimes, data quality, or recovery time metrics. Similarly,because DieCast controls workload generation it is ap-propriate for considering a variety of attack conditions.For instance, it can be used to subject an Internet serviceto large-scale Denial-of-Service attacks. DieCast mayenable evaluation of various DOS mitigation strategiesor software architectures.

Many difficult-to-isolate bugs result from system con-figuration errors (e.g., at the OS, network, or applicationlevel) or inconsistencies that arise from “live upgrades”of a service. The resulting faults may only manifest aserrors in a small fraction of requests and even then aftera specific sequence of operations. Operator errors andmis-configurations [22,24] are also known to account fora significant fraction of service failures. DieCast makes itpossible to capture the effects of mis-configurations andupgrades before a service goes live.

At the same time, DieCast will not be appropriatefor certain service configurations. As discussed earlier,DieCast is unable to scale down the memory or storagecapacity of a service. Services that rely on multi-petabytedata sets or saturate the physical memories of all of theirmachines with little to no cross-machine memory/storageredundancy may not be suitable for DieCast testing. Ifsystem behavior depends heavily on the behavior of theprocessor cache, and if multiplexing multiple VMs ontoa single physical machine results in significant cache pol-lution, then DieCast may under-predict the performanceof certain application configurations.

DieCast may change the fine-grained timing of indi-vidual events in the test system. Hence, DieCast may notbe able to reproduce certain race conditions or timing er-rors in the original service. Some bugs, such as memory

leaks, will only manifest after running for a significantperiod of time. Given that we inflate the amount of timerequired to carry out a test, it may take too long to isolatethese types of errors using DieCast.

Multiplexing multiple virtual machines onto a singlephysical machine, running with an emulated network,and dilating time will introduce some error into the pro-jected behavior of target services. This error has beensmall for the network services and scenarios we evalu-ate in this paper. In general however, DieCast’s accuracywill be service and deployment-specific. We have notyet established an overall limit to DieCast’s scaling abil-ity. In separate experiments not reported in this paper, wehave successfully run with scaling factors of 100. How-ever, in these cases, the limitation of time itself becomessignificant. Waiting 10 times longer for an experimentto configure is often reasonable, but waiting 100 timeslonger becomes difficult.

Some services employ a variety of custom hardware,such as load balancing switches, firewalls, and storageappliances. In general, it may not be possible to scalesuch hardware in our test environment. Depending onthe architecture of the hardware, one approach is to wrapthe various operating systems for such cases in scaled vir-tual machines. Another approach is to run the hardwareitself and to build custom wrappers to intercept requestsand responses, scaling them appropriately. A final optionis to run such hardware unscaled in the test environment,introducing some error in system performance. Our workwith PanFS shows that it is feasible to scale unmodifiedservices into the DieCast environment with relatively lit-tle work on the part of the developer.

7 Related WorkOur work builds upon previous efforts in a number ofareas. We discuss each in turn below.

Testing scaled systemsSHRiNK [25] is perhaps mostclosely related to DieCast in spirit. SHRiNK aims toevaluate the behavior of faster networks by simulat-ing slower ones. For example, their “scaling hypothe-sis” states that the behavior of 100Mbps flows througha 1Gbps pipe should be similar to 10Mbps through a100Mbps pipe. When this scaling hypothesis holds, itbecomes possible to run simulations more quickly andwith a lower memory footprint. Relative to this effort, weshow how to scale fully operational computer systems,considering complex interactions among CPU, network,and disk spread across many nodes and topologies.

Testing through Simulation and Emulation Onepopular approach to testing complex network services isthrough building a simulation model of system behaviorunder a variety of access patterns. While such simula-tions are valuable, we argue that simulation is best suitedto understanding coarse-grained performance character-

istics of certain configurations. Simulation is less suitedto configuration errors or to capturing the effects of un-expected component interactions, failures, etc.

Superficially, emulation techniques (e.g. Emulab [34]or ModelNet [31]), offer a more realistic alternative tosimulation because they support running unmodified ap-plications and operating systems. Unfortunately, suchemulation is limited by the capacity of the available phys-ical hardware and hence is often best suited to consider-ing wide-area network conditions (with smaller bisectionbandwidths) or smaller system configurations. For in-stance, multiplexing 1000 instances of an overlay across50 physical machines interconnected by Gigabit Ether-net may be feasible when evaluating a file sharing ser-vice on clients with cable modems. However, the same50 machines will be incapable of emulating the networkor CPU characteristics of 1000 machines in a multi-tiernetwork service consisting of dozens of racks and high-speed switches.

Time Dilation DieCast leverages earlier work on TimeDilation [19] to assist with scaling the network configura-tion of a target service. This earlier work focused on eval-uating network protocols on next-generation networkingtopologies, e.g., the behavior on TCP on 10Gbps Ether-net while running on 1Gbps Ethernet. Relative to thisprevious work, DieCast improves upon time dilation toscaledowna particular network configuration. In addi-tion, we demonstrate that it is possible to trade time forcompute resources while accurately scaling CPU cycles,complex network topologies, and disk I/O. Finally, wedemonstrate the efficacy of our approach end-to-end forcomplex, multi-tier network services.

Detecting Performance AnomaliesThere have beena number of recent efforts to debug performance anoma-lies in network services, including Pinpoint [14], Mag-Pie [9], and Project 5 [8]. Each of these initiatives an-alyzes the communication and computation across mul-tiple tiers in modern Internet services to locate perfor-mance anomalies. These efforts are complementary toours as they attempt to locate problems in deployed sys-tems. Conversely, the goal of our work is to test particu-lar software configurations at scale to locate errors beforethey affect a live service.

Modeling Internet ServicesFinally, there have beenmany efforts to model the performance of network ser-vices to, for example, dynamically provision them in re-sponse to changing request patterns [16,30] or to rerouterequests in the face of component failures [12]. Onceagain, these efforts typically target already running ser-vices relative to our goal of testing service configura-tions. Alternatively, such modeling could be used to feedsimulations of system behavior or to verify at a coarsegranularity DieCast performance predictions.

8 ConclusionTesting network services remains difficult because oftheir scale and complexity. While not technically or eco-nomically feasible, a comprehensive evaluation wouldrequire running a test system identically configured toand at the same scale as the original system. Such test-ing should enable finding performance anomalies, failurerecovery problems, and configuration errors under a vari-ety of workloads and failure conditions before triggeringcorresponding errors during live runs.

In this paper, we present a methodology and frame-work to enable system testing to more closely matchboth the configuration and scale of the original system.We show how to multiplex multiple virtual machines,each configured identically to a node in the original sys-tem, across individual physical machines. We then di-late individual machine resources, including CPU cycles,network communication characteristics, and disk I/O, toprovide the illusion that each VM has as much comput-ing power as corresponding physical nodes in the orig-inal system. By trading time for resources, we enablemore realistic tests involving more hosts and more com-plex network topologies than would otherwise be pos-sible on the underlying hardware. While our approachdoes add necessary storage and multiplexing overhead,an evaluation with a range of network services, includ-ing a commercial filesystem, demonstrates our accuracyand the potential to significantly increase the scale andrealism of testing network services.

AcknowledgementsThe authors would like to thank Tejasvi Aswatha-narayana, Jeff Butler and Garth Gibson at Panasas fortheir guidance and support in porting DieCast to theirsystems. We would also like to thank Marvin McNett andChris Edwards for their help in managing some of the in-frastructure. Finally, we would like to thank our shep-herd Steve Gribble, and our anonymous reviewers fortheir time and insightful comments—they helped tremen-dously in improving the paper.

References[1] BitTorrent. http://www.bittorrent.com.

[2] FreeBSD bootloader stops with BTX halted in hvmdomU. http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=622.

[3] Netem. http://linux-net.osdl.org/index.php/Netem.

[4] Panasas.http://www.panasas.com.

[5] Panasas ActiveScale Storage Cluster Will Provide I/O forWorld’s Fastest Computer.http://panasas.com/press_release_111306.html.

[6] RUBiS. http://rubis.objectweb.org.

[7] Vmware appliances. http://www.vmware.com/vmtn/appliances/.

[8] M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds,and A. Muthitacharoen. Performance Debugging for Dis-tributed Systems of Black Boxes. InProceedings of the19th ACM Symposium on Operating System Principles,2003.

[9] P. Barham, A. Doelly, R. Isaacs, and R. Mortier. UsingMagpie for Request Extraction and Workload Modelling.In Proceedings of the 6th USENIX Symposium on Oper-ating Systems Design and Implementation, 2004.

[10] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris,A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen andthe Art of Virtualization. InProceedings of the 19th ACMSymposium on Operating System Principles, 2003.

[11] L. A. Barroso, J. Dean, and U. Holzle. Web Search fora Planet: The Google Cluster Architecture.IEEE Micro,2003.

[12] J. M. Blanquer, A. Batchelli, K. Schauser, and R. Wol-ski. Quorum: Flexible Quality of Service for Internet Ser-vices. InProceedings of the 3rd USENIX Symposium onNetworked Systems Design and Implementation, 2005.

[13] E. Cecchet, J. Marguerite, and W. Zwaenepoel. Perfor-mance and scalability of EJB applications. InProceedingsof the 17th ACM Conference on Object-Oriented Pro-gramming, Systems, Languages and Applications, 2002.

[14] M. Y. Chen, E. Kiciman, E. Fratkin, A. Fox, andE. Brewer. Pinpoint: Problem Determination in Large,Dynamic Internet Services. InProceedings of the 32nd In-ternational Conference on Dependable Systems and Net-works, 2002.

[15] Y.-C. Cheng, U. Hoelzle, N. Cardwell, S. Savage, andG. M. Voelker. Monkey See, Monkey Do: A Tool for TCPTracing and Replaying. InProceedings of the USENIXAnnual Technical Conference, 2004.

[16] R. Doyle, J. Chase, O. Asad, W. Jen, and A. Vahdat.Model-Based Resource Provisioning in a Web ServiceUtility. In Proceedings of the USENIX Symposium on In-ternet Technologies and Systems, 2003.

[17] G. R. Ganger and contributors. The DiskSim Simu-lation Environment. http://www.pdl.cmu.edu/DiskSim/index.html.

[18] D. Gupta, K. V. Vishwanath, and A. Vahdat. DieCast:Testing Network Services with an Accurate 1/10 ScaleModel. Technical Report CS2007-0910, University ofCalifornia, San Diego, 2007.

[19] D. Gupta, K. Yocum, M. McNett, A. C. Snoeren, G. M.Voelker, and A. Vahdat. To Infinity and Beyond: Time-Warped Network Emulation. InProceedings of the 3rdUSENIX Symposium on Networked Systems Design andImplementation, 2006.

[20] A. Haeberlen, A. Mislove, and P. Druschel. Glacier:Highly durable, decentralized storage despite massivecorrelated failures. InProceedings of the 3rd USENIXSymposium on Networked Systems Design and Implemen-tation, 2005.

[21] J. Mogul. Emergent (Mis)behavior vs. Complex SoftwareSystems. InProceedings of the first EuroSys Conference,2006.

[22] K. Nagaraja, F. Oliveira, R. Bianchini, R. P. Martin, andT. D. Nguyen. Understanding and Dealing with OperatorMistakes in Internet Services. InProceedings of the 6thUSENIX Symposium on Operating Systems Design andImplementation, 2004.

[23] W. Norcott and D. Capps. IOzone Filesystem Benchmark.http://www.iozone.org/.

[24] D. Oppenheimer, A. Ganapathi, and D. Patterson. Whydo Internet services fail, and what can be done about it. InProceedings of the 4th USENIX Symposium on InternetTechnologies and Systems, 2003.

[25] R. Pan, B. Prabhakar, K. Psounis, and D. Wischik.SHRINK: A Method for Scalable Performance Predictionand Efficient Network Simulation. InIEEE INFOCOM,2003.

[26] R. Ricci, C. Alfeld, and J. Lepreau. A Solver for theNetwork Testbed Mapping Problem. InSIGCOMM Com-puter Counications Review, volume 33, 2003.

[27] L. Rizzo. Dummynet and Forward Error Correction. InProceedings of the USENIX Annual Technical Confer-ence, 1998.

[28] The MPI Forum. MPI: A Message Passing Interface.pages 878–883, Nov. 1993.

[29] A. Tridgell. Emulating Netbench.http://samba.org/ftp/tridge/dbench/.

[30] B. Urgaonkar, P. Shenoy, and T. Roscoe. Resource over-booking and application profiling in shared hosting plat-forms. InProceedings of the 5th USENIX Symposium onOperating Systems Design and Implementation, 2002.

[31] A. Vahdat, K. Yocum, K. Walsh, P. Mahadevan, D. Kostic,J. Chase, and D. Becker. Scalability and Accuracy in aLarge-Scale Network Emulator. InProceedings of the 5thUSENIX Symposium on Operating Systems Design andImplementation, 2002.

[32] C. A. Waldspurger. Memory Resource Management inVMware ESX Server. InProceedings of the 5th USENIXSymposium on Operating Systems Design and Implemen-tation, 2002.

[33] A. Warfield, R. Ross, K. Fraser, C. Limpach, andH. Steven. Parallax: Managing Storage for a Million Ma-chines. InProceedings of the 10th Workshop on Hot Top-ics in Operating Systems.

[34] B. White, J. Lepreau, L. Stoller, R. Ricci, S. Guruprasad,M. Newbold, M. Hibler, C. Barb, and A. Joglekar. An In-tegrated Experimental Environment for Distributed Sys-tems and Networks. InProceedings of the 5th USENIXSymposium on Operating Systems Design and Implemen-tation, 2002.