Evaluating Distributed Systems: Does Background Trafﬁc …...Evaluating Distributed Systems: Does Background Trafﬁc Matter? Kashi Venkatesh Vishwanath and Amin Vahdat University

Evaluating Distributed Systems: Does Background Traffic Matter?

Kashi Venkatesh Vishwanath and Amin VahdatUniversity of California, San Diego{kvishwanath,vahdat}@cs.ucsd.edu

AbstractEvaluating novel networked protocols and services re-

quires subjecting the target system to realistic Internetconditions. However, there is no common understand-ing of what is required to capture such realism. Conven-tional wisdom suggests that competing background traf-fic will influence service and protocol behavior. Onceagain however, there is no understanding of what as-pects of background traffic are important and the extentto which services are sensitive to these characteristics.

Earlier work shows that Internet traffic demonstratessignificant burstiness at a range of time scales. Unfor-tunately, existing systems evaluations either do not con-sider background traffic or employ simple synthetic mod-els, e.g., based on Poisson arrivals, that do not capturethese burstiness properties. In this paper, we show thatrealistic background traffic, has qualitatively differentimpact on application and protocol behavior than sim-ple traffic models. One conclusion from our work isthat applications should be evaluated under a range ofbackground traffic characteristics to determine the rela-tive merits of applications and to understand behavior incorner cases of live deployment.

1 IntroductionThere has been significant interest in understanding thecharacteristics of network services and protocols underrealistic deployment conditions [2, 3, 4, 13]. Applicationdevelopers typically have two high-level options whenevaluating their prototype: live deployment or emula-tion/simulation. Live deployment on a testbed such asPlanetLab allows developers to subject their system to re-alistic network conditions and failures. However, exper-iment management/control and reproducibility becomemore difficult. Further, while the deployment environ-ment is realistic, there is no guarantee that it is represen-tative or that any series of experiments will experience arange of desired potential network conditions.

Emulation and simulation on the other hand simplifyexperiment management and make it relatively easy toobtain reproducible results. However, while recent em-ulation environments [15, 20] allow unmodified applica-tions, they must still simulate some target network condi-tions, including topology, failure characteristics, routing,and background traffic. Practitioners are left with the un-

enviable task of developing appropriate models for eachof these important network characteristics. Thus, whileemulation offers the promise of evaluating applicationsunder a range of conditions, the huge space of potentialconditions makes it difficult, if not impossible, for mostdevelopers to take advantage of this flexibility.

One long term goal of our work is to enable develop-ers to use emulation to evaluate their applications under arange of scenarios with realistic models of traffic, topol-ogy, routing, and host characteristics. We leave a studyof the relative sensitivity of applications to different as-pects of the network, e.g., routing protocols versus trafficcharacteristics [21, 19] versus topology [7, 11], to futurework. Our goal in this paper is to understand applica-tion sensitivity to background traffic. It is clear that ap-plications behave differently when competing with othertraffic. Thus, it is not surprising that researchers havebegun to include background traffic models in their eval-uations (see Section 2 for a summary). However, it isalso well-known that Internet traffic has rich and com-plex properties not captured by models for backgroundtraffic, e.g., self-similarity and burstiness at a range oftimescales [8, 22]) not captured by the simple modelsemployed by practitioners. A natural question to answerthen, is whether these complex but realistic models, war-rant attention as candidates for background traffic.

In this paper, we quantify application sensitivity to arange of background traffic characteristics. That is, aresimple models of background traffic, such as constant bitrate, Poisson arrivals, or deterministic link loss rates, suf-ficient to capture the effects of background traffic? Ordo we require more complex background traffic mod-els that capture the burstiness on a particular networklink? We begin with a literature survey to understandthe common techniques for modeling background traf-fic. We also leverage recent work [19, 21] on modelingand recreating background traffic characteristics for ex-isting Internet links. Using accurate, real-time networkemulation, we subject a number of applications to a spec-trum of background traffic models and report variationsin end-to-end application behavior.

We find qualitative differences in application behav-ior when employing simple synthetic models of back-ground traffic rather than realistic traffic models. Weinvestigate the cause of this difference and present our

USENIX ’08: 2008 USENIX Annual Technical ConferenceUSENIX Association 227

initial findings. Specifically, we find that differences inburstiness between two background traffic models at arange of timescales significantly impacts overall appli-cation behavior, even when network links are not con-gested. Existing synthetic models for background trafficdo not demonstrate the rich variances in the packet arrivalprocess for competing traffic present in live network traf-fic. Thus, studies employing these synthetic models maymischaracterize the impact of background traffic. Fur-ther, we find that even seemingly small differences in theburstiness of background traffic for realistic traffic mod-els can lead to important differences in application be-havior. Hence, one conclusion of this work is that studiesof network application behavior should include experi-ments with a range of realistic background traffic mod-els. Ideally, the research community would evolve a suiteof background traffic models representative of a range ofnetwork conditions and locations. We hope that our find-ings in the paper will serve as a means to spur sufficientinterest in the community to collectively develop such anappropriate benchmark suite in the future.

In summary, this paper makes the following contribu-tions. This work is the first to quantify the impact ofa range of background traffic characteristics on a num-ber of applications. Prior to this work, it was not pos-sible to deterministically subject application traffic to arange of realistic network conditions while accountingfor the complexity of real network traffic, e.g., as deter-mined by TCP. We present a methodology for doing soand use this methodology to carry out a systematic sen-sitivity study of applications to a range of network char-acteristics. We show that techniques such as replaying apre-existing trace packet-by-packet do not exhibit the re-sponsiveness of real Internet traffic. Similarly, we showthat common models for generating background traffic,such as transmitting traffic at a constant bit rate, trafficwith a Poisson arrival process, or deterministically set-ting loss rates to network links has significantly less im-pact on application traffic than realistic Internet traffic.

Investigating the cause of these observations, our de-tailed performance evaluation shows that the propertiesof Internet traffic, in particular its burstiness across arange of time scales, can have unpredictable impact ona range of applications relative to simpler traffic models.We establish that it is not enough to simply use “some”bursty source as a background traffic model. As anotherexample, we reproduce the results of an earlier study thatemployed synthetic traffic models to compare bandwidthestimation tools. After validating the original results, wefound that some of the conclusions of the earlier studymay have been reversed when employing realistic trafficmodels.

2 Motivation and Related WorkConsider the problem of determining the sensitivity of agiven application to a range of background traffic char-acteristics. One approach would simply be to run theapplication across the Internet on a testbed such as Plan-etLab. Unfortunately, the difficulty to measure the char-acteristics of background traffic at any point in time iscompounded by the fact that one cannot guarantee re-producing a particular set of background traffic charac-teristics. Finally, experiments would be restricted to thetype of background traffic experienced in a particular de-ployment scenario, making it difficult to extrapolate tosensitivity in other settings.

Hence, a careful study of application sensitivity tobackground traffic must run in an emulation or simula-tion environment prior to live deployment. This begs thequestion as to what kind of background traffic to employ.One approach is to take a trace of background traffic ata particular point in the Internet and to replay that traf-fic packet-by-packet in an emulation environment. Aswe quantify later, such background traffic will not be re-sponsive to the application traffic. It will blindly trans-mit data in a preconfigured sequence; real Internet traf-fic responds and adapts to prevailing traffic conditions asa result of end-to-end congestion control. Other simpleapproaches involve playing back traffic at a constant bitrate, according to a Poisson arrival process, or using de-terministic loss rates. Unfortunately, these simple tech-niques are known not to reproduce the characteristics ofInternet traffic and, as we quantify later, will result inincorrectly estimating the impact of background traffic.

In this paper, we present a methodology for quantify-ing the impact of realistic Internet traffic on a range of ap-plications. We build on our earlier work [21] that showshow to create traffic that is both realistic and responsive.By realistic, we mean that the traffic matches the com-plex characteristics of traffic across some original link,including traffic burstiness at a range of timescales. Byresponsive, we mean that the background traffic adaptsto application traffic in the same manner that they wouldin the wild. That is, the flows in aggregate ramp up andrecover from loss in a similar manner that they wouldacross the Internet, e.g., as determined by TCP’s re-sponse to round trip times, bottleneck bandwidths, etc.

Critical to our methodology are techniques to repro-duce the application- and user-level characteristics of theflows in some original trace, e.g., session initiation ac-cording to user behavior, packet sizes according to proto-col behavior, etc. We also recreate the bandwidths, laten-cies, and loss rates observed in the original trace. Repro-ducing these network conditions is important to enablingresponsiveness of our generated background traffic to thecharacteristics of the foreground/application traffic.

USENIX ’08: 2008 USENIX Annual Technical Conference USENIX Association228

Explanation (%) Project/Paper Title and the Conference NameNo Background Traffic 25.6 SIGCOMM ’06 - Churn in distributed systems, SpeakUp,

SIGCOMM ’04 - Modeling P2P, Mercury, OSDI ’04 - FUSE, NSDI ’05- Quorum, Lowbandwidth DHT routing, NSDI ’04 - Macedon, Thor-SS,SIGCOMM ’07 - Structured streams, NSDI ’07 - SET

Constant Bit Rate Traffic 2.33 SIGCOMM ’04 - CapProbeFixed Loss Rate 2.33 OSDI ’04-FUSELowered Link Capacity 2.33 NSDI ’06 - DOTOnly Latencies 2.33 NSDI ’06-Colyseus“Some” Background Flows 2.33 NSDI ’05 - Trickles“Some” TCP source 2.33 SIGCOMM ’04 - CapProbeCustom Built Simulator 4.65 NSDI ’05 - Myths about structured/unstructured overlays,GlacierPareto Flow Arrival 4.65 SIGCOMM ’05 - VCP, NSDI ’06 - PCPFixed Length Flows 2.33 NSDI ’06 - PCPLong Lived flows 4.65 SIGCOMM ’05 - TFRC, SIGCOMM ’07 - PERTLRD Traffic 2.33 SIGCOMM ’04 - CapProbePareto Length Flows 2.33 NSDI ’06 - PCPSpecWeb 6.98 OSDI ’06 - TCP offload, NSDI ’06 - Connection conditioning, NaKika.Run on PlanetLab 18.6 NSDI ’06 - CoBlitz, OASIS, NaKika. NSDI ’05 - Shark, Botz4Sale,

NSDI ’04 - Saxons, NSDI ’07 - BitTyrant, SETReal World Deployment 9.3 NSDI ’06 - Overcite, NSDI ’04 - BAD-FS, TotalRecall, NSDI ’07 - BitTyrantHarpoon 2.33 SIGCOMM ’05 - End-to-end loss rate measurementRON Testbed 2.33 NSDI ’05 - MONET

Table 1: Literature survey of SIGCOMM, SOSP/OSDI and NSDI from 2004-2007.

Background. To motivate the importance of back-ground traffic for a range of studies, we conducted a lit-erature survey of SIGCOMM, SOSP/OSDI and NSDIfrom 2004-2007. We determined whether each papercontained a performance evaluation of a distributed sys-tem and, if so, what types of background traffic were em-ployed in the evaluation. Overall, we found 35 papersthat conducted a total of 43 such experiments. Table 1summarizes a subset of these experiments, along with adescriptive project name and the publication venue. Wedivide the set of techniques into four main categories.

Our study is vulnerable to sampling bias, howeverwe make the following high-level observations. Morethan 25% (11/43) of the experiments use no backgroundtraffic (NBG). The application to be evaluated is typi-cally run on a cluster of machines with high-speed in-terconnect. Another 14% of the experiments accountfor congestion using simple models such as constantbit rate (CBR) traffic or simply constraining link laten-cies/capacities in a synthetic topology.

At the other extreme (bottom of Table 1), approxi-mately 30% of the experiments employ live deploymenton testbeds such as PlanetLab, RON, and Harpoon. Fi-nally, in the middle of the table we have 25% of exper-iments that are done with some sophisticated models toaccount for background traffic, for instance, Caprobe ex-periments use Long Range Dependent (LRD) traffic.

Based on this study, we observe significant confusionin the community regarding whether background traf-fic is an important consideration in carrying out exper-imental evaluations. Further, there is no consensus as

to what type of background traffic should be employed.Finally, in virtually all (29/35) papers just one model ofbackground traffic is used with no analysis of applicationsensitivity to different background traffic conditions; thiscan partially be attributed to the large space of possiblemodels of traffic.

A goal of our work is to enable the community tomake informed decisions about whether background traf-fic should be considered in a particular scenario, andif so, the particular characteristics of background traf-fic that are important to consider. Thus, we consider theinteraction of a variety of applications with a range ofcompeting background traffic. Ideally, we would con-sider all of the background traffic models summarized inTable 1 against all of the 43 experiments that we foundin those papers. In this paper, we take a few steps to-ward this goal. For instance, we show that CBR andPoisson traffic have very similar impact on the applica-tions we consider and that setting probabilistic loss ratesdoes not capture the complexity of real interactions withbackground traffic. We also study the impact of realisticbackground traffic models on application performance.

Related Work. WASP [14] is perhaps most closely re-lated to our work in spirit. It shows that HTTP perfor-mance can be significantly impacted by setting realis-tic delays and loss rates for the flows. The work con-cludes that web services cannot be evaluated on highspeed LANs, but must instead consider wide-area net-working effects. Relative to this effort, we considera number of application classes and background traffic


conditions. We show that simple models of wide-areaconditions (such as higher round trip times and non-zeroloss rates) are insufficient to capture the effects of realis-tic Internet background traffic either.

Our work will benefit from ongoing work in produc-ing realistic Internet traffic, including Tmix [9], Har-poon [19], Surge [6], and Swing [21]. We chose to gen-erate realistic background traffic using Swing, but we ex-pect qualitatively similar results had we employed alter-native tools. We make no claims, positive or negative,about whether the traffic we generate is realistic or not.Rather, we consider a range of qualitatively and quanti-tatively different traffic conditions and show the resultingeffect on application performance. However, we do liketo add that before the advent of Swing, it was not possi-ble to create realistic and responsive network traffic in atestbed environment [21]. This partially explains why re-searchers have been using ad-hoc traffic models in theirexperiments to date.

3 MethodologyWe begin by describing the architecture used for carryingout the experiments followed by the list of applicationswe used. We then describe the background traffic modelsused followed by the experiments we conducted.

3.1 Architecture

MultimediaClientApache Webserver

StreamingMedia Server

Web Browser

Dumb−bell Link

Background TrafficBackground TrafficSources/Sinks Sources/SinksSwing/Harpoon/

SURGE/Poisson

Figure 1: Evaluation architecture.

Figure 1 depicts our approach to quantifying and un-derstanding the impact of background traffic on individ-ual applications. We place traffic sources and sinks oneither side of a constrained/dumb-bell link such that alltraffic in both directions crosses the common link. Twoclasses of sources and sinks generate the traffic cross-ing the link. In the first class depicted at the bottom,nodes generate application traffic, for instance, ApacheWeb server and httperf clients. Then on top, we havesources and sinks responsible for generating backgroundtraffic for the target link, for instance using Swing [21],Harpoon [19], SURGE [6] or simpler traffic sources suchas Poisson or Constant Bit Rate (CBR). Because appli-

cation traffic and background traffic share the commonlink, we can quantify the impact on the application as afunction of a range of background traffic characteristics.

In a real network environment, applications must com-pete with background traffic at multiple links betweenthe source and destination. However, there are no knowntechniques to model desired background traffic charac-teristics at multiple successive links in some larger topol-ogy. For instance, there may be strong correlations be-tween the background traffic characteristics of links inthe topology. For the purposes of this study, we feelit reasonable to quantify and understand the impact ofbackground traffic at a single link before attempting toextrapolate to more complex scenarios. In all likelihood,the effect of background traffic at multiple links will beeven more pronounced than our findings. Hence, our re-sults should be interpreted as a conservative estimate ofthe effects of background traffic, while still demonstrat-ing application sensitivity to varying background trafficcharacteristics.

3.2 ApplicationsOf course, the impact of background traffic heavily de-pends upon the characteristics of the particular applica-tion under consideration. For this study, we chose threeapplications with diverse communication patterns and re-quirements: Web traffic, multimedia streaming, and end-to-end bandwidth estimation. Note that each of these ap-plications exercise one end-to-end path, and hence, weexplicitly omit more distributed applications such as Bit-Torrent that simultaneously exercise multiple indepen-dent Internet paths. While this class of application is im-portant, considering complex topologies is beyond thescope of this paper (see above). We did run experi-ments (not discussed further here) for the case where allBitTorrent clients were subject to a dumbbell topology;these results were qualitatively similar to our findings forHTTP.

Web Traffic For Web applications, we set out to de-termine the effect of background traffic on the responsetimes perceived by end clients. We placed a singleApache Web server on one side of the dumbbell (e.g.,the bottom left in Figure 1). We programmed httperfclients to fetch objects of various sizes from the serverand placed them on the other side of the dumbbell. Thelinks connecting the clients and server to the dumbbellhave large capacity and low latency, such that the dumb-bell is the constrained link for all client-server commu-nication (we vary the capacity of the dumbbell link invarious experiments described below).

To generate background traffic, we place sources andsinks of the appropriate traffic generator on either sidesof the target link (top left/right in Figure 1). We set thebandwidths, latencies, and loss rates of the links connect-


Trace Secs Trace BW Trace Collection Number Dominant Unique↓ Aggregate Dir0 Date of flows applications IPs

(Mbps) (Mbps) (1000s)Auck 600 5.5 3.3 June 11, 2001 155 K HTTP, SQUID 3Mawi 900 17.8 7.8 September 23, 2004 476 K HTTP, RSYNC 15

Mawi2 900 11.9 10.8 December 30, 2003 160 K HTTP, NNTP 8

Table 2: Trace characteristics for three different links.

ing background traffic sources and sinks based on thetraffic generation model. For instance, we simply playback CBR traffic over unconstrained links; whereas forSwing, we assign assign latencies, bandwidths and lossrates based on observed path characteristics in some orig-inal packet trace [21].

Multimedia Traffic The second application class weconsider is video streaming. Video clients are sensi-tive to the arrival times of individual packets, whereasweb clients are typically sensitive to end-to-end transfertimes. Overall, we wish to quantify the impact of vari-ous types of background traffic on client-perceived videoquality. For streaming audio/video we use the free ver-sion of Helix on the server side and Real Player on theclient side. We generate background traffic across thedumbbell topology as with Web traffic.

Bandwidth Estimation Tool We chose bandwidth es-timation for our third application. While not an endapplication, it displays fundamentally different charac-teristics than our first two applications and is a build-ing block for many higher-level services. We employPathload [10], and pathChirp [17] tools for our study.We place bandwidth senders and receivers along withcompeting traffic generators across the dumbbell topol-ogy identically to our configuration for Web and videostreaming.

3.3 Traffic GenerationWe consider four techniques for generating competingbackground traffic, in increasing order of complexity andrealism. First, for constant bit rate (CBR) traffic, wewrote simple sources to generate packets at a specifiedrate to sinks on the opposite side of the dumbbell link.In aggregate, the sources generate a target overall rateof background traffic. Second, for Poisson traffic, wemodify the sources to generate traffic with byte arrivalper unit time governed by a Poisson process with a givenmean. We evaluated variants of CBR and Poisson usingboth UDP and TCP transports.

While CBR and Poisson processes do not capture thecomplexities of real Internet traffic [16], we wish toquantify the resulting differences in end-to-end applica-tion behavior relative to more realistic, but complex traf-fic generation techniques. Hence, for our third technique,we modify the sources and sinks to play back packets in

the exact pattern specified by an available tcpdump oftraffic across an existing Internet link (Table 2). Onedrawback of this approach is that the generated back-ground traffic is not congestion responsive. That is, thetraffic will be played back in exactly the same pattern asthe original trace irrespective of the behavior of the ap-plication traffic. Another drawback is that it is difficultto extrapolate to alternative, but similar, scenarios whenplaying back a fixed trace (e.g., changing the availablebandwidth across the constrained link, the distribution ofround trip times between sources and sinks, etc.).

Thus, for our fourth technique, we use Swing [21] togenerate responsive and realistic network traffic. Swingis a closed-loop, network responsive traffic generator thatqualitatively captures the packet interactions of a rangeof applications using a simple structural model. Startingfrom a packet trace, Swing automatically extracts distri-butions for user, application, and network behavior. Itthen generates live packet traffic corresponding to theunderlying models in a network emulation [15, 20] en-vironment running commodity network protocol stacks.Because Swing generates the traffic using real TCP/UDPstacks, the resulting traffic is responsive both to fore-ground traffic and varying characteristics of the con-strained link and end-to-end loss rates/round trip times.

Swing extracts a range of distributions from an orig-inal trace to model user behavior, e.g., think time be-tween requests, application behavior, e.g., distribution ofrequest and response sizes, and network characteristics.One particularly important aspect of network traffic thatSwing is able to reproduce is burstiness in the packet ar-rival process at a range of time scales [21]. Doing sorequires Swing to assign latencies, bandwidths, and lossrates to the links leading to and from the shared link inour evaluation architecture based on the observed distri-bution of round trip times, link capacities, and loss ratesin some original trace. This way, TCP flows ramping upto their fair share bandwidth or recovering from loss willdo so with statistically similar patterns (including bursti-ness in aggregate) as in the original trace conditions.

An additional benefit of employing high-level mod-els of user, application, and network characteristics forbackground traffic is that it becomes possible to mod-ify certain model parameters to extrapolate to alternativescenarios [21]. For instance, when reducing the round


5 10 15

10

15

20

25

30

Time Scale j

log 2(E

nerg

y(j))

Smallest time scale = 1 msec [ BG traffic ] (Bytes)

HighBurst [3.4173 Mbps]Auck [3.4435 Mbps]Poisson [3.3470 Mbps]CBR [3.3610 Mbps]

Figure 2: Auck-based background traffic (Energy plot).

0 100 200 300 400 500 6000

5

10

15HighBurst

0 100 200 300 400 500 6000

5

10

15

Ban

dwid

thin

1se

cbi

n(M

bps)

Auck

0 100 200 300 400 500 6000

5

10

15Poisson

Figure 3: Auck-based background traffic (Time Series).

trip times for flows, overall burstiness tends to increasebecause flows tend to increase their transmission ratesmore quickly. We refer to alternate background trafficgenerated by perturbing distributions of various parame-ters in Swing as variants of the original trace.

We run all our experiments, including the sourcesand sinks for both foreground (Web, multimedia, band-width estimation) and background traffic (CBR, Poisson,Playback, Swing) in the ModelNet emulation environ-ment [20]. For our experiments, we configure all sourcesand sinks of both foreground and background traffic toroute all of their packets through ModelNet. Briefly,ModelNet subjects each packet to the per hop bandwidth,delay, queueing, and loss characteristics of the targettopology by inspecting both the source and destination ofthe packet and routing it through the emulated topology.ModelNet operates in real time, meaning that it movespacket from queue to queue in the target topology beforeforwarding it on to the destination machine assuming thatthe packet was not dropped. Earlier work [20] validatesModelNet’s accuracy using a single traffic shaper at traf-fic rates up to 1Gbps (we can operate at higher speeds byemploying multiple traffic shapers in parallel).

3.4 Topology and Experiments

We run our experiments on a cluster of commodity work-stations, multiplexing multiple instances on each work-station depending on the requirements. For the exper-iments in this paper, we use eight 2.8 Ghz Xeon pro-cessors running Linux 2.6.10 (Fedora Core 2) with 1GBmemory and integrated Gigabit NICs. We generate back-ground traffic using 1000 nodes in the emulated topology(meaning that we multiplex hundreds of emulated nodesonto each physical machine).

For example, for a 200Mbps trace, assuming an evensplit between generators and listeners (four machineseach), each generator would be responsible for accu-rately initiating flows corresponding to 50Mbps on av-erage. Each machine can comfortably handle the aver-age case, though there are significant bursts that makeit important to “over-provision” the experimental infras-

tructure. To avoid contention between physical resourceswith the traffic generator we use a separate set of ma-chines to host the target application; for instance, httperfclients. In this fashion, any performance impact and vari-ations that we measure results purely from network be-havior.

We first describe the traces we use to generate realis-tic background traffic for the experiments in this paper.We use traces from Mawi [12], a trans-Pacific line, aswell as an OC3c ATM link traces from the University ofAuckland, New Zealand [5]. These traces come fromdifferent geographical locations (New Zealand, Japan)and demonstrate variation in application mix, averagethroughput, number of users etc. (see Table 2). All traceswere taken on 100Mbps links. The Mawi traces wereconstrained to 18Mbps over long time periods (though itcould burst higher).

While all original and the corresponding Swing-generated traces are bidirectional, we focus on the im-pact of competing traffic in one direction of traffic (Dir0in Table 2) for simplicity in our plotted results. In otherwords, we design all experiments such that the dominantdirection of application traffic (HTTP responses, video,bandwidth estimation packet train) matches Dir0 of thegenerated background traffic.

We use wavelet-scaling plots [1, 8, 22] to character-ize traffic burstiness. Intuitively, these plots allow visualinspection of burstiness for a range of timescales. The x-axis of these plots shows the time scale on a log scaleand the y-axis shows the corresponding energy value.Higher levels of energy correspond to more burstiness.Figure 2 plots burstiness corresponding to five variantsof the Auck trace for a 20Mbps link (along with the av-erage bandwidth for each variant in square brackets). Weconfigured Swing to generate constant-bitrate (CBR) andPoisson traffic with the same average bandwidth as theAuckland (Auck) trace. In addition to reproducing theAuck trace using Swing, we also created a very burstytraffic variant, called HighBurst (HB), by setting theround trip times artificially to 4ms while generating traf-fic using Swing. The lower round trip times (relative to


CT CU PT PU TPB UPB M2 HB0

100

200

300

400

500

Varying Burstiness/Responsiveness

resp

.tim

e(%

inc

inm

ean

vs.N

BG

)

Congestion Responsiveness

CBR Poisson

TrafficPlayback

Figure 4: “Simple” models are inaccurate.

600 800 1000 1200 1400 16000

0.2

0.4

0.6

0.8

1

Time to finish one download (msecs)

Cum

ulat

ive

frac

tion

LossRate 1%LossRate 2%LossRate 3%Mawi2

Figure 5: Deterministic loss rates never suffice.

the distribution in the original trace) means that TCP traf-fic ramps up more quickly, recovers more quickly fromlosses, etc. For an alternate visualization of the relativedifference in burstiness, consider the time series plots forthe same traffic models shown in Figure 3. The Poissonvariant generates a relatively fixed bandwidth whereasHighBurst variant peaks to 15Mbps at times, comparedto Auck. Note that while the average bandwidth con-sumption for all traces are comparable, finer-grained be-havior varies significantly. Further, none of the variantscome close to congesting a 20 Mbps link.

4 ResultsWe subject each of our three target application classes tothe various types of background traffic described abovewith the goal of answering the following questions:i) What aspects of “realism” should we reproduce? Is itsufficient to simply “replay” individual packets in somemeasured trace or does the generated background trafficneed to be TCP responsive and react appropriately basedon the end-to-end network characteristics of the trafficsources and sinks? (§ 4.1)ii) Can probabilistic packet drops susbstitute for realcompeting background traffic? (§ 4.2)iii) Does burstiness of background traffic matter or is itsufficient to reproduce average bandwidth? (§ 4.2, 4.4)iv) Are some applications more sensitive to backgroundtraffic than others? (§ 4.3)v) Is application behavior sensitive to slight variations inthe background traffic characteristics? (§ 4.4)

4.1 Background Traffic ResponsivenessThe first question we consider is the importance of re-alistically playing back background traffic characteris-tics. We consider three techniques for doing so: i)scheduling per-packet transmissions using UDP connec-tions to match the exact timings (at 1 ms granularity) andpacket arrival processes found in some original trace; ii)scheduling per-packet transmissions using a single TCPconnection to attempt to match the packet arrival processfound in the original trace; and iii) extracting user, ap-plication, and network characteristics from the original

trace and playing back TCP flows whose communica-tion patterns are statistically similar to the original with-out making an attempt to match the patterns found in theoriginal trace.

The first technique (UDP) is not responsive to thecharacteristics of foreground traffic. Thus, it will notback off in the face of congestion. The second technique(TCP) is responsive, but, unfortunately it becomes im-possible to perform precise packet scheduling for TCPconnections. Further, because we have no knowledge ofthe end-to-end network characteristics for the TCP flowswe play back, it is not possible to verify that the responseto congestion would match the behavior of flows fromthe original trace (e.g., because of variations in roundtrip times or losses elsewhere in the network). Finally,because it employs a single connection to multiplex thebehavior of a much larger number of flows, its behav-ior is unpredictable. The third technique, correspondingto Swing-based [21] playback, promises to be the mostfaithful but is also the most complex and requires moreresources (i.e., logic to source and sink traffic from indi-vidual hosts) for trace playback.

To establish the accuracy for each of these techniques,we run httperf clients requesting 1 MB files from anApache Web server sharing the bottleneck link with theMawi2 trace. We set the shared link (the point of con-tention between httperf and background traffic) to 15Mbps. We choose 15 Mbps to ensure that the backgroundtraffic attempts to consume a significant portion of avail-able resources (§ 4.2 onwards relaxes this assumption).We are interested in understanding the response time forHTTP as a function of the characteristics of the back-ground traffic. As described earlier, the links connectingthe httperf/Apache nodes to the shared link are uncon-strained (large capacity and low latency), so that trafficshaping takes place only at the target link. During eachexperiment we fetch files back-to-back, using a singleclient-server pair for 10 minutes.

Figure 4 shows the results for different classes of back-ground traffic. For each scenario, we plot the mean andstandard deviation of response time increase relative tothe NBG (No Background Traffic) case. Background


400 600 800 1000 12000

0.2

0.4

0.6

0.8

1


Cum

ulat

ive

frac

tion

Auck, Poisson, Largefile, 20Mbps Link

NBG [0 Mbps]Poisson [3.3 Mbps]Poisson2 [6.6 Mbps]Poisson4 [13.2 Mbps]

Figure 6: Poisson background traffic.

500 1000 1500 20000

0.2

0.4

0.6

0.8

1


Cum

ulat

ive

frac

tion

Auck, Swing, Largefile, 20Mbps Link

NBG [0 Mbps]Swing [3.3 Mbps]SwingRSP2 [6.6 Mbps]SwingRSP4 [13.2 Mbps]

Figure 7: Swing background traffic.

traffic consumes 66% of the shared link’s capacity in allcases. The first two bars plot slowdown for CBR trafficwith both TCP and UDP playback (CT and CU) con-figured to consume as much average bandwidth as theMawi2 trace. The next two bars show slowdown for TCPand UDP variants of Poisson (PT and PU). For both ofthe models (CBR and Poisson) UDP variants impactedresponses time much more (200% increase vs 100%increase) than TCP variants exposing the limitation ofUDP based traffic generators. Because UDP sources arenot congestion responsive, they have a much larger im-pact on HTTP. The next two bars show the slowdown forTCP- and UDP-based playback (TPB/UPB) of Mawi2.TPB has much less impact than Swing-based playbackof Mawi2 (M2), likely because its use of a single TCPconnection means that the generated background trafficis less bursty. Finally, UPB results in larger slowdownthan Swing-based playback because it is not congestionresponsive.

Given the above results, we conclude that simple tech-niques for “playing back” background traffic, such asUBP and TBP, may result in significant inaccuracy asthe aggregate traffic across a link approaches the link’scapacity. Thus, for the remainder of this paper, we em-ploy Swing to play background traffic corresponding tosome original network condition and compare it to othervariants such as CBR/Poisson.

Another popular technique in the literature for captur-ing the complexity of real background traffic is to setloss rates for particular links, with the goal of captur-ing the effects of losses caused by competing traffic. Todetermine whether this technique could capture the ef-fects of more complex traffic scenarios, we next mea-sure the performance of httperf when setting various lossrates for the shared link. In all other respects, this ex-periment is identical to the case where we run with nobackground traffic. Figure 5 shows that httperf’s behav-ior when crossing a link with a range of loss rates differsfrom its behavior when competing with realistic Mawi2traffic, again, across a 15Mbps bottleneck link. For in-stance, with losses of 1% the CDF of retrieval times istoo far to the left of Mawi2. On the other hand, higher

loss rate settings, while shifting the CDF to the right, in-crease the size of the tail too. We hypothesize that thedifference in behavior results from the independent na-ture of losses when setting any fixed loss rate (relative tothe bursty losses common for Internet links) and the factthat setting loss rate alone does not capture any of theeffects of increased queueing delay (and hence increasedround trip times) leading up to the point of loss.

4.2 Httperf/Apache

Having established the appropriate technique for trans-mitting background traffic, we now turn our attention tounderstanding the impact of background traffic on HTTPtransfers. In order to arrive at a conservative estimateof the impact of background traffic we first experimentwith the Auck trace (lowest bandwidth and least bursty).We begin by examining the impact of varying band-widths of background traffic (based on Auck) on httperfperformance. We fetch 1 MB files across a 20Mbpslink and vary the load placed by background traffic bygenerating traffic for three different average throughputs(3.3, 6.6, and 13.2 Mbps) corresponding to variants ofthe Auck trace. In the baseline (3.3 Mbps) case, weemploy Swing parameterized by the original Auck trace.In the alternative cases, we modify the distribution ofresponse sizes such that the average bandwidth increasesby a factor of 2 and 4 (traces SwingRSP2/SwingRSP4),resulting in average bandwidths of 6.6 Mbps and 13.2Mbps respectively. We compare the impact of Swing-generated traffic to those of TCP-generated CBR andPoisson traffic of the same average bandwidth.

Figures 6 and 7 show the CDFs of download timescorresponding to various bandwidth/burstiness combina-tions for background traffic. Along with the legend namewe show in square brackets the average bandwidth of thebackground traffic for reference. The effects of CBR andPoisson traffic are similar, so we only plot the results forPoisson (relative to Poisson, the CBR curves are virtu-ally vertical with the same median value). As shown inFigure 6, the impact of Poisson (and hence CBR) trafficis almost entirely predicted by the average level of back-


0 20 40 60 80 1000

50

100

150

200

250

300

350

Link Capacity (Mbps)

%in

crea

sein

dow

nloa

dtim

efr

omN

BG

Auck Variants, Largefile

Auck 90th percentileAuck medianPoisson 90th percentilePoisson median

Figure 8: Varying link capacities.

5 %ile median 95 %ile0

0.5

1

1.5

Res

pons

eT

ime

(sec

)

Different Traces (20Mbps)

3.8 secs

AuckAuckHBMawiMawiHBMawi2Mawi2HB

Figure 9: Simply generating “bursty” traffic is not enough.

ground traffic bandwidth. The distribution of downloadtimes shifts to the right, but with little variation in re-sponse times as the average level of background trafficincreases. The Swing-generated Auck trace has a morevaried impact on application performance (Figure 7).With low levels of competing traffic (3.3 Mbps), the dis-tribution of download times is similar to Poisson. Alsoof interest is the fact that the performance for the 5th-percentile of retrievals is actually faster for Auck thanfor Poisson traffic for both the 6.6 Mbps (SwingRSP2vs. Poisson2) and the 13.2 Mbps (SwingRSP4 vs. Pois-son4) bandwidths. In these cases, the flows were luckyto be subject to less competing traffic than their counter-parts in the Poisson trace. Bursty traffic means that peri-ods of high activity coexist with periods of lower activity.Moving to higher levels of average utilization, the curvesbecome significantly skewed. For instance, the 90th per-centile of download time for the Auck (AuckRSP4) traceat 13.2 Mbps is 1484 ms compared to 759 ms for Poisson(Poisson4) traffic.

Thus, less bursty background traffic means that per-formance is governed by the average amount of avail-able bandwidth and, expectedly, there is relatively smallvariation in download times across individual object re-trievals. When background traffic is steady, HTTP per-formance is predictable. As burstiness increases, themean download time increases, as does the variation inperformance. Some flows can get lucky, behaving almostas if there is no background traffic; while others may beunlucky with significantly worse performance than themean.

Next, we consider the sensitivity of HTTP perfor-mance to background traffic characteristics as a functionof the fraction of shared link capacity occupied by thebackground traffic. That is, it could be the case thatbackground traffic characteristics (e.g., burstiness) areonly important when consuming a significant fraction oflink capacity. Figure 8 shows the impact on downloadtimes for varying levels of background traffic with av-erage bandwidths of 3.3Mbps (same as the Auck trace).The y-axis shows the slowdown (median as well as 90th-percetile) of HTTP transfers of large 1 MB files relative

to the NBG case as a function of the capacity (5-100Mbps) of the shared link on the x-axis. On the left sideof the graph we have cases corresponding to a highly uti-lized link; while on the right side, the background traf-fic consumes a small fraction of overall capacity. Forlarge files and low levels of link utilization, the graphsshow that burstiness of background traffic does not mat-ter. For instance, for a 50Mbps link the impact on me-dian response time is independent of the burstiness; theslowdown largely corresponds to the fraction of the linkconsumed by the background traffic.

When background traffic consumes a significant por-tion of link capacity however, it is sufficient to cause sig-nificant losses for the HTTP transfers. Thus, at the 90thpercentile of slowdown, we find that transfers competingwith bursty traffic completed significantly more slowly,around a factor of 1.5 for Auck, than with a less burstytraffic source, i.e., Poisson. However, unlike median re-sponse times, this relative ordering is present even whenoverall capacity is high (e.g., 100 Mbps). Thus, for largefiles, burstiness of traffic matters at all levels of link uti-lization, but more so at high levels of utilization.

The impact on transfer times for different file sizes asa function of different levels of burstiness of backgroundtraffic is futher explored in § 5.2. Overall we concludethat impact on download time for web transfers is a func-tion of the size of the download and the average bandwithof background traffic as well as its burstiness.

Finally, we consider whether it is sufficient to sim-ply playback a “bursty” traffic source or whether traf-fic sources with different burstiness characteristics willhave different impacts on HTTP performance. Thus, weconsidered the impact of six different bursty backgroundtraffic characteristics competing for a shared 20 Mbpslink. We considered background traffic corresponding toAuck, Mawi, and Mawi2. We further modified each ofthese sources to be high burst variants (“HB”) by settingthe round trip times for all flows to 4msec while generat-ing traffic using Swing. We plot the slowdown of HTTPtransfers (1MB file) relative to the NBG case at the 5th,median, and 95th percentile in Figure 9.

There are a number of interesting results here. First,


10−1

100

101

1020

0.2

0.4

0.6

0.8

1

Packet jitter in milliseconds

Cum

ulat

ive

frac

tion

NBG 6MbpsAuck 6MbpsAuck 4MbpsAuck 3.5Mbps

Figure 10: Varying capacity.

10−1

100

101

1020

0.2

0.4

0.6

0.8

1

Packet jitter in milliseconds

Cum

ulat

ive

frac

tion

NBG 4MbpsAuck 4MbpsNobuff 4MbpsAuck 4Mbps HB

Figure 11: Varying burstiness and client-buffering.

the Mawi2 trace appears to have a larger impact on HTTPthan Mawi, even though it consumes significantly lessaggregate bandwidth (see § 5 for explanation). Further,the HB-variant of Mawi2 has significantly less impacton HTTP performance at the 5th and 50th percentile,but more than a factor of 5 more impact at the 95th per-centile. Its highly bursty nature means that a significantnumber of flows are lucky and suffer comparable slow-down to the less bursty cases. However, a number offlows are extremely unlucky and suffer large slowdowns.The tail for the Mawi2HB case is significantly longer aswell, an undesirable character for HTTP transfers wherethe humans in the loop typically value predictable be-havior. This experiment shows that although burstinessof traffic is important to consider, simply reproducing“some” bursty traffic is not enough.

4.3 Multimedia

We next consider the impact of background traffic onmultimedia traffic. We aim to understand the impactas a function of the capacity of the link, the burstinessof the generated traffic as well as the amount of client-buffering. We run the publicly available Helix Server toserve real media content to RealPlayer. As with all ourexperiments, the application traffic competes with vari-ous types of background traffic at the shared link. Likehttperf, we consider 1 client-server pair. We tested withvarious streaming rates but the results in this paper are fora 450Kbps CBR stream encoded using RealVideo codecat 30 frames per second that runs for 62 seconds.

One important question for multimedia traffic is the“metric of goodness” for a multimedia stream, whichshould correspond to the quality of the video playedback. However, developing such a metric based on thedata stream received at clients is challenging. Thus, as aproxy for such a quality metric we use a range of statis-tics based on the stream delivered to the client and cor-roborate it with visual inspections. For a 450 kbps videostream, we can roughly assume that receiving more than450 kbps of instantaneous bandwidth results in accept-able video quality at that particular point in time.

Another important metric of interest is jitter, the time

between successive packets received by the client. Lowerlevels of jitter will correspond to higher video quality.For our first experiment, we run with Auck backgroundtraffic (3.3 Mbps average) and compare its impact on jit-ter for shared links with 3.5 Mbps, 4 Mbps, and 6 Mbpscapacity(Realplayer was insensitive to background traf-fic at higher levels of link capacity). Figure 10 shows thedistribution of inter-packet timing (jitter) for this experi-ment. We also perform an experiment in the absence ofany background traffic (NBG) for baseline. For a 6 Mbpstarget link, background traffic does not change the qual-ity of video (inspected by viewing the video) whereasfor a 4 Mbps link the degradation is moderate. It isonly when we constrain the link capacity to 3.5 Mbpsthat video quality suffers significantly. One conclusion isthat unless we are operating in extreme scenarios (avail-able bandwidth approximately equal to the bandwidth ofthe stream), RealPlayer is relatively insensitive to burstybackground traffic.

To test this hypothesis, we next attempt to stress thelimits of RealPlayer. We modify the client-side imple-mentation to reduce the amount of buffering from a de-fault of 10 seconds to 0 seconds. Figure 11 plots thedistribution of jitter with and without buffering for the4Mbps link. With 10 seconds of buffering, backgroundtraffic had no impact on jitter (Auck 4Mbps). However,there is significant impact (verified by visual inspection)when we remove the buffering. Without buffering, thereal server attempts to retransmit lost packets more ag-gressively in order to meet real time deadlines, consum-ing more network resources and negatively impactingoverall jitter. With sufficient buffering, the server can af-ford to be more relaxed about retransmissions, resultingin an overall smoother transmission rate.

Finally, we consider the highburst background trafficsource by setting RTTs in Swing to 4msec. In this case,flows ramp up and down very quickly causing bursty traf-fic on the shared link. Figure 11 also plots the distribu-tion of jitter corresponding to this experiment. The resultshows that even for tight links and bursty backgroundtraffic the performance degradation in realplayer is mod-erate (again verified by visual inspection).


0

20

40

60

80

100

Ban

dwid

thE

stim

ate

(Mbp

s)

BG 30MbpsBG 50Mbps

BG 70Mbps

CBRPois

sM

awi

HBCBR

Poiss

Maw

iHB

CBRPois

sM

awi

HB

Actual pChirp Pload − low Pload − high

Figure 12: Augmenting to and validating pathload vs. pathchirp experiments from [17].

Overall, we conclude that for RealPlayer, unless weare operating in regimes of low buffering, low avail-able bandwidth or very high burstiness, any reasonablebackground traffic model should suffice in performanceevaluations for systems similar to RealPlayer; in mostcases, RealPlayer’s default buffering will likely result inacceptable quality of service for a range of backgroundtraffic characteristics, as long as sufficient average band-width levels are available. Interestingly, RealPlayer’s(successful) buffering and retransmission scheme is in-formed by significant experience with live Internet de-ployments subject to a range of bursty competing crosstraffic.

4.4 Bandwidth Estimation

For our final experiments, we consider the sensitivity ofbandwidth estimation tools to background traffic charac-teristics. We use Pathload [10] and pathChirp [17] for ourstudy because they are publicly available and because re-cent studies indicate that they are among the most accu-rate for bandwidth estimation [18]. To determine the sen-sitivity of existing tools to a range of background trafficcharacteristics, we repeat experiments from earlier pub-lished work comparing the relative merits of Pathload topathChirp [17]. In these experiments, the authors em-ployed Poisson and CBR models for background traffic.Below, we show that at least the conclusions from thisearlier work may be reversed if the experiments had con-sidered more realistic background traffic characteristics.

For our experiments, we overprovision all links suchthat the available bandwidth measured for the path is de-termined by the bandwidth available on the shared linkin our model topology. We set Pathload’s timeout to fiveminutes, and we report the average and standard devi-ation for the low and high estimates of available band-width across 25 runs. In practice, a small spread betweenthe low and high estimates of available bandwidth and alow standard deviations for multiple reported values re-flects likely accurate bandwidth estimates. PathChirp, onthe other hand, periodically outputs an estimate of the

available bandwidth and does not distinguish between alow and a high value.

Given our exact knowledge of the generated back-ground traffic, we also calculate the true values of theavailable bandwidth for one second bins. In the graphsthat follow, we plot the percentage error reported by eachbandwidth estimation tool relative to our calculated val-ues for available bandwidth.

The experiments from [17] compare available band-width across a 100Mbps link as measured by pathloadand pathchirp against various flavors of background traf-fic. They employ CBR UDP traffic and UDP Poissontraffic to create three different scenarios with availablebandwidths of 30, 50 and 70 Mbps. Figure 12 showsthese results. Additionally, we also playback Mawi, anda HighBurst variant of Mawi (labeled HB) for the threelevels of available bandwidth. In each case, we increasethe number of user-sessions (in Swing) by an appropri-ate value to match the levels of bandwidth consumed byCBR and Poisson traffic. For instance, to get the avail-able bandwidth of 70mbps, we multiplied the number ofsessions for Mawi by 3.3 times.

We initially discuss the results for Poisson and CBRmodels, reproducing the earlier experiments [17]. First,we confirm the earlier result [17] that when backgroundtraffic is low (BG 30Mbps) or moderate (BG 50Mbps)the estimates of pathload as well as pathchirp are close tothe actual values. We also observe that indeed pathchirptakes 10-20% the amount of bytes consumed by pathloadto arrive at similar estimates (not shown here). However,we note that a couple of results differed. For instance,pathchirp overestimated the bandwidth for a heavily uti-lized link, ie. when background traffic was around 70Mbps. Similarly, for low link utilization (BG 30Mbps),pathchirp is slightly less accurate than pathload.

We next move to the effects of bursty background traf-fic, not considered by the earlier work, the bars corre-sponding to Mawi and HB in Figure 12. We make a num-ber of new interesting observations here. First, the resultsfor both tools degrade when competing with more bursty


170 180 190 200 2100

0.2

0.4

0.6

0.8

1


Cum

ulat

ive

frac

tion

100mbps, Large file

NBGMawiMawi2

(a) 100Mbps Link

400 600 800 1000 12000

0.2

0.4

0.6

0.8

1


Cum

ulat

ive

frac

tion

20mbps, Large file

NBGMawiMawi2

(b) 20Mbps Link

Figure 13: Significance of traffic in the reverse direction.

traffic sources, indicated by either an inaccuracy in themean or large standard deviation. PathChirp appears tobe a bit less sensitive to burstiness of background traf-fic. We also no longer see that pathchirp is always con-servative in its estimates. In fact, when the backgroundtraffic occupies a significant portion of the link we seethat pathchirp overestimates available bandwidth, e.g.,Mawi/HB traffic with 70 Mbps background traffic. Theauthors [17] saw that both pathchirp and pathload esti-mates are close to each other, however, we see that thereare times when the estimates are very different from eachother; for instance, Mawi for 50 Mbps.

The conclusion from these results is that it is difficultto predict a priori which bandwidth estimation tool givesbetter results and that realistic network traffic charac-teristics can impact study results. System behavior is afunction of the average amount of background traffic aswell as burstiness. For instance, if we hypothesize thatonly background traffic’s average throughput is impor-tant, then we can disprove it by observing the case for 30Mbps BG traffic. For CBR background traffic, pathloadis the more accurate tool, whereas for HB, pathchirp ismore accurate in its estimate. Similarly if we hypothe-size that only burstiness matters, then we can disprovethat by looking at HB traffic. At 30Mbps we would pre-fer pathchirp for HB traffic whereas at 70Mbps pathloadshows superior accuracy. We leave the task of attributingthe sensitivity of these tools to the underlying algorithmsto orthogonal future work.

5 Case Studies

Considering our evaluation to this point, sensitivity tobackground traffic characteristics is application specific.Certain applications, such as bandwidth estimation arehighly sensitive to background traffic while others, suchas RealPlayer, are relatively insensitive except in extremecases. We now turn our attention to some additional im-portant characteristics of background traffic and their im-pact on end-to-end applications.

5.1 Bi-directional Traffic Characteristics

To this point, we have largely focused on traffic charac-teristics in one direction of a link (though in all cases,we played back bi-directional traces). We now considera case where traffic characteristics in the reverse direc-tion can impact application performance depending onthe deployment setting. In the first experiment, we re-peat retrievals of 1MB files using httperf/Apache (as Sec-tion 4.2) across a shared 100 Mbps target link. We usethe background traffic models corresponding to the twoMawi traces in Table 2. The traces are in increasing orderof bandwidth in the direction of flow of HTTP responses(Dir 0). One would expect that the response time dis-tribution would correspond to the relative bandwidth ofeach of these traces.

Figure 13a) plots the CDFs of retrieval times for thisexperiment. Mawi2 is of higher bandwidth than Mawi(Table 2) but the CDF is to the left of Mawi as a resultof background traffic in the reverse path. This effect ismore pronounced for Mawi as it has 10 Mbps of trafficin the reverse direction versus 1 Mbps for Mawi2. Thisrelative ordering, however, is not present when we repeatthe experiments with a 20 Mbps shared link as shownin Figure 13b). At 20 Mbps, the forward direction traf-fic dominates for both Mawi and Mawi2 so the effectsof congestion on the reverse path is less pronounced.Thus, one simple conclusion is that that background traf-fic in the reverse direction can impact application perfor-mance though the dominant direction is difficult to pre-dict a priori.

5.2 Burstiness at Various Timescales

We have shown that background traffic with the same av-erage bandwidth, but differing burstiness characteristicswill have varying impact on application behavior. Wenow consider the question of whether burstiness at par-ticular timescales (for instance, burstiness at millisecondversus second granularity) has differing impact on appli-cation performance. Generating traffic that selectivelyand precisely varies burstiness at arbitrary timescales isan open problem. However, we can alter burstiness in


0 5 10 15

18

20

22

24

26

28

30

Time Scale j

log 2(E

nerg

y(j))

Traffic 1; Think times = 0AuckTraffic 2; Link Capacites = 2Mbps

(a) Burstiness of background traffic models

1000 1500 2000 25000

0.2

0.4

0.6

0.8

1


Cum

ulat

ive

frac

tion

Large file

Traffic 1AuckTraffic 2

(b) Large files affected by large timescales

0 20 40 60 800

0.2

0.4

0.6

0.8

1


Cum

ulat

ive

frac

tion

Small file

Traffic 1AuckTraffic 2

(c) Small files affected by small timescales

Figure 14: Varying burstiness at small and large timescales to understand impact on file download times.

a coarse manner at relatively small and large timescales.To vary burstiness at large timescales, we set Swing’suser think time distribution to 0 for all requests; mean-ing, for instance, that for each user session, requestsfor subsequent objects will be initiated immediately af-ter the previous request completes. Figure 14a) showsthe resulting energy plot. Note that traffic becomes morebursty at timescales 11−14 (corresponding to the 1-8 secrange) relative to the default Auck trace. To vary bursti-ness at timescales 3− 6 (4-32 ms), we restrict all links inthe emulated topology for Swing sources to 2 Mbps. Fig-ure 14a) also shows the energy plot for this case. Let uscall these two new traces Traffic1 and Traffic2. Wewill next try to understand if these seemingly small dif-ferences in burstiness impacts application performance.

We run HTTP experiments across a shared 10 Mbpslink. The results in Figure 14b) shows the effect of vary-ing burstiness on a relatively relatively large 1 MB filetransfer. We see that varying burstiness at small timescales (Traffic2) has relatively little impact on the dis-tribution of response times. Burstiness at larger timescales (Traffic1) skews the CDF around the median.Few flows finish early and few take longer compared toAuck/Traffic2. We hypothesize that over a long transfer,burstiness at small time scales (milliseconds) averagesout over the lifetime of the connection. Burstiness overmultiple seconds however will more significantly impacttransfers that complete in just over one second by default.

The situation reverses itself when we consider the dis-tribution of download times for the same experiment butwith 4 KB objects as shown in Figure 14c). In thiscase, increased burstiness at large timescales (Traffic1)has no impact on the distribution of performance rela-tive to the baseline. However, the decreased burstiness atsmall timescales for Traffic2 relative to Auck results inimproved download times, especially above the 80th per-centile. Note that Traffic2 displays reduced burstiness inthe 4-32 ms timescales (Figure 14a), and that retrievinga 4 KB takes 29 ms in the median case for Auck. Fromthis initial experiment, we hypothesize that for a given

level of background traffic, its burstiness characteristicsat timescales corresponding to the duration of individ-ual application operations will have the most significantimpact on overall application behavior.

6 Discussion

While this paper shows the importance of subjecting net-work services to a range of background traffic conditions,there are a number of remaining open questions. Firstis the need for a suite of background traffic that capturethe range of conditions likely to be experienced duringlive deployment. We have shown that the bursty trafficpresent on the Internet can dramatically affect applica-tion behavior relative to synthetic traffic models. How-ever, we cannot yet characterize the full range of networkconditions likely to be present on the Internet.

Next, we have not yet shown the best way to gener-ate the background traffic. We have shown one plausibletechnique employing the Swing traffic generator. How-ever, Swing requires multiple machines to generate thetraffic and a network emulator to appropriately shape in-dividual flows. And yet, our current understanding indi-cates that recreating Internet traffic burstiness criticallydepends on recreating appropriate network characteris-tics and closed-loop responsive traffic sources and sinksthat, for example, obey the dynamics of TCP [21].

Finally, our analysis considers the effects of back-ground traffic on a single link between sources and des-tinations. Under realistic deployment scenarios, appli-cation traffic must interact with background traffic atmultiple links across the network. Our initial experi-ments, not shown for brevity, indicate that the impact ofbursty background traffic is even more pronounced andunpredictable when considering more complex networktopologies. We believe that capturing this full complex-ity will be challenging in the short term, but it would bevaluable to determine whether relatively simple modelscan account for most of the additional impact that comesfrom more complex topologies.


7 ConclusionWe set out to answer a simple question: When run-ning simulation or emulation experiments, what kind ofbackground traffic models should be employed? Ad-ditional motivation comes from recent interest in accu-rately recreating realistic background traffic characteris-tics. While there have been significant advancements inthis space, there is relatively little understanding of whataspects of background traffic actually impact applicationbehavior. To fill this gap, we quantified the interactionof applications with a variety of background traffic mod-els. We found that, for instance, HTTP is sensitive to theburstiness of background traffic depending on the domi-nant size of transferred objects; multimedia applicationshave been engineered to be relatively insensitive to traf-fic burstiness; and bandwidth estimation tools are highlysensitive to bursty traffic because unstable link character-istics make convergence to stable estimates difficult. Wealso observed that characteristics of background traffic inboth directions of a link can impact application perfor-mance. Finally, we hypothesize that each application issensitive to burstiness of traffic at particular application-dependent timescales.

AcknowledgementsWe would like to thank the reviewers for their insight-ful and detailed comments that helped improve both thequality and presentation of this paper. Ryan Braud, ChrisEdwards and Marvin McNett helped set up the infras-tructure behind our experiments.

References[1] ABRY, P., AND VEITCH, D. Wavelet Analysis of Long-

range-dependent Traffic. IEEE Transactions on Informa-tion Theory (1998).

[2] ALAN SHIEH AND ANDREW C. MYERS AND EMIN

GUN SIRER. Trickles: A Stateless Network Stack for Im-proved Scalability, Resilience, and Flexibility. In NSDI(2005).

[3] ANDERSON, T., COLLINS, A., KRISHNAMURTHY, A.,AND ZAHORJAN, J. PCP: Efficient Endpoint CongestionControl. In NSDI (2006).

[4] ANNAPUREDDY, S., FREEDMAN, M. J., AND

MAZIERES, D. Shark: Scaling File Servers viaCooperative Caching. In NSDI (2005).

[5] Auckland-VII Trace Archive, University of Auckland,New Zealand. http://pma.nlanr.net/Traces/long/auck7.html.

[6] BARFORD, P., AND CROVELLA, M. Generating Repre-sentative Web Workloads for Network and Server Perfor-mance Evaluation. In MMCS (1998).

[7] CHEN, J., GUPTA, D., VISHWANATH, K. V., SNOEREN,A. C., AND VAHDAT, A. Routing in an Internet-Scale

Network Emulator. In Proceedings of the IEEE/ACM In-ternational Symposium on Modeling, Analysis, and Sim-ulation of Computer and Telecommunication Systems(MASCOTS ’04) (October 2004).

[8] FELDMANN, A., GILBERT, A. C., HUANG, P., AND

WILLINGER, W. Dynamics of IP Traffic: A Study of theRole of Variability and the Impact of Control. In ACMSIGCOMM (1999).

[9] HERNANDEZ-CAMPOS, F., SMITH, F. D., AND JEFFAY,K. Generating Realistic TCP Workloads. In CMG2004Conference (2004).

[10] JAIN, M., AND DOVROLIS, C. Pathload: A Measure-ment Tool for End-to-end Available Bandwidth. In PAM(2002).

[11] MAHADEVAN, P., HUBBLE, C., HUFFAKER, B., KRI-OUKOV, D., AND VAHDAT, A. Orbis: Rescaling DegreeCorrelations to Generate Annotated Internet Topologies.In ACM SIGCOMM (August 2007).

[12] MAWI Working Group Traffic Archive. http://tracer.csl.sony.co.jp/mawi/.

[13] NAGARAJA, K., OLIVEIRA, F., BIANCHINI, R., MAR-TIN, R. P., AND NGUYEN, T. D. Understanding andDealing with Operator Mistakes in Internet Services. InOSDI (2004).

[14] NAHUM, E. M., ROSU, M.-C., SESHAN, S., AND

ALMEIDA, J. The Effects of Wide-area Condi-tions on WWW Server Performance. In SIGMET-RICS/Performance (2001).

[15] OF UTAH, U. Emulab.Net: The Utah Network Testbed.http://www.emulab.net.

[16] PAXSON, V., AND FLOYD, S. Wide-Area Traffic: TheFailure of Poisson Modeling. In IEEE/ACM Transactionson Networking (1995).

[17] RIBEIRO, V., RIEDI, R., BARANIUK, R., NAVRATIL, J.,AND COTTRELL, L. pathChirp: Efficient Available Band-width Estimation for Network Paths. In PAM (2003).

[18] SHRIRAM, A., MURRAY, M., , HYUN, Y., BROWNLEE,N., BROIDO, A., FOMENKOV, M., AND KC CLAFFY.Comparison of Public End-to-End Bandwidth EstimationTools on High-Speed Links. In PAM 2005.

[19] SOMMERS, J., AND BARFORD, P. Self-Configuring Net-work Traffic Generation. In IMC (2004).

[20] VAHDAT, A., YOCUM, K., WALSH, K., MAHADEVAN,P., KOSTIC, D., CHASE, J., AND BECKER, D. Scalabil-ity and Accuracy in a Large-Scale Network Emulator. InOSDI (2002).

[21] VISHWANATH, K. V., AND VAHDAT, A. Realistic andResponsive Network Traffic Generation. In ACM SIG-COMM (2006).

[22] WILLINGER, W., PAXSON, V., AND TAQQU, M. S. Self-similarity and Heavy Tails: Structural Modeling of Net-work Traffic. In A Practical Guide to Heavy Tails: Statis-tical Techniques and Applications (1998).


Documents

Evaluating Distributed Systems: Does Background Trafﬁc …...Evaluating Distributed Systems: Does Background Trafﬁc Matter? Kashi Venkatesh Vishwanath and Amin Vahdat University